首页 Paddle框架 帖子详情
Error: Cannot find original variable with name ifelse_inputifelse_3_7@GRAD
收藏
快速回复
Paddle框架 问答深度学习 7384 2
Error: Cannot find original variable with name ifelse_inputifelse_3_7@GRAD
收藏
快速回复
Paddle框架 问答深度学习 7384 2
  • 环境
    1)PaddlePaddle版本:1.6.2
    2)CPU:Intel(R) Xeon(R) CPU E5-2620 v2
    3)GPU:no
    4)系统环境:CentOS 6.3 ,Python版本 3.6.5

  • 问题描述
    程序直接 core 掉了,报了个“Cannot find original variable with name ifelse_inputifelse_3_7@GRAD”的错,哪里梯度传导出了问题?没有显式的打印“Python Call Stacks (More useful to users)”

I0116 15:18:22.781353  1479 parallel_executor.cc:421] The number of CPUPlace, which is used in ParallelExecutor, is 1. And the Program will be copied 1 copies
I0116 15:18:22.813120  1479 build_strategy.cc:363] SeqOnlyAllReduceOps:0, num_trainers:1
I0116 15:18:22.855751  1479 parallel_executor.cc:285] Inplace strategy is enabled, when build_strategy.enable_inplace = True
I0116 15:18:22.879025  1479 parallel_executor.cc:368] Garbage collection strategy is enabled, when FLAGS_eager_delete_tensor_gb = 1
/home/xxx/lib/python3.6/site-packages/paddle/fluid/executor.py:779: UserWarning: The following exception is not an EOF exception.
  "The following exception is not an EOF exception.")
Traceback (most recent call last):
  File "script/text2sql_train.py", line 45, in <module>
    trainer.train_and_eval()
  File "/home/xxx/text2sql/textone/training/trainer.py", line 249, in train_and_eval
    metrics_tensor_value = self.train_exe.run(fetch_list=self.train_metrics_tensor_name)
  File "/home/xxx/lib/python3.6/site-packages/paddle/fluid/parallel_executor.py", line 311, in run
    return_numpy=return_numpy)
  File "/home/xxx/lib/python3.6/site-packages/paddle/fluid/executor.py", line 780, in run
    six.reraise(*sys.exc_info())
  File "/home/xxx/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/home/xxx/lib/python3.6/site-packages/paddle/fluid/executor.py", line 775, in run
    use_program_cache=use_program_cache)
  File "/home/xxx/lib/python3.6/site-packages/paddle/fluid/executor.py", line 834, in _run_impl
    return_numpy=return_numpy)
  File "/home/xxx/lib/python3.6/site-packages/paddle/fluid/executor.py", line 674, in _run_parallel
    tensors = exe.run(fetch_var_names)._move_to_list()
paddle.fluid.core_avx.EnforceNotMet:

--------------------------------------------
C++ Call Stacks (More useful to developers):
--------------------------------------------
0   std::string paddle::platform::GetTraceBackString<char const*>(char const*&&, char const*, int)
1   paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int)
2   paddle::framework::Scope::RenameInternal(std::string const&, std::string const&) const
3   paddle::framework::Scope::Rename(std::string const&) const
4   paddle::operators::ConditionalBlockGradOp::AssignLocalGradientToGlobal(paddle::platform::Place const&, paddle::framework::Scope const&, std::string const*, unsigned long, std::vector<std::string, std::allocator<std::string> > const&)
const
5   paddle::operators::ConditionalBlockGradOp::RunImpl(paddle::framework::Scope const&, paddle::platform::Place const&) const
6   paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, paddle::platform::Place const&)
7   paddle::framework::details::ComputationOpHandle::RunImpl()
8   paddle::framework::details::OpHandleBase::Run(bool)
9   paddle::framework::details::FastThreadedSSAGraphExecutor::RunOpSync(paddle::framework::details::OpHandleBase*)
10  paddle::framework::details::FastThreadedSSAGraphExecutor::RunOp(paddle::framework::details::OpHandleBase*, std::shared_ptr<paddle::framework::BlockingQueue<unsigned long> > const&, unsigned long*)
11  std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, void> >::_M_invoke(std::_Any_data const&)
12  std::__future_base::_State_base::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&, bool&)
13  ThreadPool::ThreadPool(unsigned long)::{lambda()#1}::operator()() const

----------------------
Error Message Summary:
----------------------
Error: Cannot find original variable with name ifelse_inputifelse_3_7@GRAD at (/paddle/paddle/fluid/framework/scope.cc:181)
0
收藏
回复
全部评论(2)
时间顺序
AIStudio786339
#2 回复于2020-01

可以打印program 看是否符合预期

0
回复
aistudio_2
#4 回复于2021-05

mark

0
回复
需求/bug反馈?一键提issue告诉我们
发现bug?如果您知道修复办法,欢迎提pr直接参与建设飞桨~
在@后输入用户全名并按空格结束,可艾特全站任一用户