首页 Paddle框架 帖子详情
已解决 EnforceNotMetTrace
收藏
快速回复
Paddle框架 问答深度学习 3846 4
已解决 EnforceNotMetTrace
收藏
快速回复
Paddle框架 问答深度学习 3846 4

训练时正常,进入测试部分后报错EnforceNotMetTraceback,求助。

代码及报错信息如下:

 

报错截图:

 

完整报错内容如下:

开始训练...

Pass 0, Step 0, Cost 0.693164, Acc 0.500000

Pass 0, Step 100, Cost 0.757676, Acc 0.437500

Pass 0, Step 200, Cost 0.687623, Acc 0.593750

Pass 0, Step 300, Cost 0.689104, Acc 0.562500

Pass 0, Step 400, Cost 0.693155, Acc 0.500000

Pass 0, Step 500, Cost 0.693761, Acc 0.406250

Pass 0, Step 600, Cost 0.693978, Acc 0.406250

EnforceNotMetTraceback (most recent call last) in ()
19 program = test_program,
20 feed = feeder.feed(data),
---> 21 fetch_list=[avg_cost, acc])
22 test_accs.append(test_acc[0])
23 test_costs.append(test_cost[0])
/opt/conda/envs/python27-paddle120-env/lib/python2.7/site-packages/paddle/fluid/executor.pyc in run(self, program, feed, fetch_list, feed_var_name, fetch_var_name, scope, return_numpy, use_program_cache)
649 scope=scope,
650 return_numpy=return_numpy,
--> 651 use_program_cache=use_program_cache)
652 else:
653 if fetch_list and program._is_data_parallel and program._program and (
/opt/conda/envs/python27-paddle120-env/lib/python2.7/site-packages/paddle/fluid/executor.pyc in _run(self, program, exe, feed, fetch_list, feed_var_name, fetch_var_name, scope, return_numpy, use_program_cache)
747 self._feed_data(program, feed, feed_var_name, scope)
748 if not use_program_cache:
--> 749 exe.run(program.desc, scope, 0, True, True, fetch_var_name)
750 else:
751 exe.run_cached_prepared_ctx(ctx, scope, False, False, False)
EnforceNotMet: Invoke operator dropout_grad error.
Python Callstacks:
File "/opt/conda/envs/python27-paddle120-env/lib/python2.7/site-packages/paddle/fluid/framework.py", line 1771, in append_op
attrs=kwargs.get("attrs", None))
File "/opt/conda/envs/python27-paddle120-env/lib/python2.7/site-packages/paddle/fluid/layer_helper.py", line 43, in append_op
return self.main_program.current_block().append_op(*args, **kwargs)
File "/opt/conda/envs/python27-paddle120-env/lib/python2.7/site-packages/paddle/fluid/layers/nn.py", line 1466, in dropout
'dropout_implementation': dropout_implementation,
File "", line 132, in net
drop7 = fluid.layers.dropout(x=fc6, dropout_prob=0.5)
File "", line 4, in
predict = model.net(input=image, class_dim=2)
File "/opt/conda/envs/python27-paddle120-env/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2878, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/opt/conda/envs/python27-paddle120-env/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2818, in run_ast_nodes
if self.run_code(code, result):
File "/opt/conda/envs/python27-paddle120-env/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2714, in run_cell
interactivity=interactivity, compiler=compiler, result=result)
File "/opt/conda/envs/python27-paddle120-env/lib/python2.7/site-packages/ipykernel/zmqshell.py", line 537, in run_cell
return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
File "/opt/conda/envs/python27-paddle120-env/lib/python2.7/site-packages/ipykernel/ipkernel.py", line 208, in do_execute
res = shell.run_cell(code, store_history=store_history, silent=silent)
File "/opt/conda/envs/python27-paddle120-env/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 399, in execute_request
user_expressions, allow_stdin)
File "/opt/conda/envs/python27-paddle120-env/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 233, in dispatch_shell
handler(stream, idents, msg)
File "/opt/conda/envs/python27-paddle120-env/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 283, in dispatcher
return self.dispatch_shell(stream, msg)
File "/opt/conda/envs/python27-paddle120-env/lib/python2.7/site-packages/tornado/stack_context.py", line 300, in null_wrapper
return fn(*args, **kwargs)
File "/opt/conda/envs/python27-paddle120-env/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 438, in _run_callback
callback(*args, **kwargs)
File "/opt/conda/envs/python27-paddle120-env/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 486, in _handle_recv
self._run_callback(callback, msg)
File "/opt/conda/envs/python27-paddle120-env/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 456, in _handle_events
self._handle_recv()
File "/opt/conda/envs/python27-paddle120-env/lib/python2.7/site-packages/tornado/stack_context.py", line 300, in null_wrapper
return fn(*args, **kwargs)
File "/opt/conda/envs/python27-paddle120-env/lib/python2.7/site-packages/tornado/ioloop.py", line 1073, in start
handler_func(fd_obj, events)
File "/opt/conda/envs/python27-paddle120-env/lib/python2.7/site-packages/ipykernel/kernelapp.py", line 499, in start
self.io_loop.start()
File "/opt/conda/envs/python27-paddle120-env/lib/python2.7/site-packages/traitlets/config/application.py", line 658, in launch_instance
app.start()
File "/opt/conda/envs/python27-paddle120-env/lib/python2.7/site-packages/ipykernel_launcher.py", line 16, in
app.launch_new_instance()
File "/opt/conda/envs/python27-paddle120-env/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/opt/conda/envs/python27-paddle120-env/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
C++ Callstacks:
Enforce failed. Expected ctx->Attrs().Get("is_test") == false, but received ctx->Attrs().Get("is_test"):1 != false:0.
GradOp is only callable when is_test is false at [/paddle/paddle/fluid/operators/dropout_op.cc:108]
PaddlePaddle Call Stacks:
0 0x7f15448e4ff8p void paddle::platform::EnforceNotMet::Init(std::string, char const*, int) + 360
1 0x7f15448e5347p paddle::platform::EnforceNotMet::EnforceNotMet(std::string const&, char const*, int) + 87
2 0x7f15455a4501p paddle::operators::DropoutOpGrad::InferShape(paddle::framework::InferShapeContext*) const + 1249
3 0x7f154684cff0p paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant const&, paddle::framework::RuntimeContext*) const + 304
4 0x7f154684d411p paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant const&) const + 529
5 0x7f154684aa0cp paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, boost::variant const&) + 332
6 0x7f1544a7146ep paddle::framework::Executor::RunPreparedContext(paddle::framework::ExecutorPrepareContext*, paddle::framework::Scope*, bool, bool, bool) + 382
7 0x7f1544a7450fp paddle::framework::Executor::Run(paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool, std::vector > const&, bool) + 143
8 0x7f15448d5f8dp
9 0x7f1544917936p
10 0x7f15c622d1a7p PyEval_EvalFrameEx + 28695
11 0x7f15c622f6c9p PyEval_EvalCodeEx + 2025
12 0x7f15c622cb98p PyEval_EvalFrameEx + 27144
13 0x7f15c622f6c9p PyEval_EvalCodeEx + 2025
14 0x7f15c622cb98p PyEval_EvalFrameEx + 27144
15 0x7f15c622f6c9p PyEval_EvalCodeEx + 2025
16 0x7f15c622f8eap PyEval_EvalCode + 26
17 0x7f15c622bcc0p PyEval_EvalFrameEx + 23344
18 0x7f15c622f6c9p PyEval_EvalCodeEx + 2025
19 0x7f15c622cb98p PyEval_EvalFrameEx + 27144
20 0x7f15c622f6c9p PyEval_EvalCodeEx + 2025
21 0x7f15c622cb98p PyEval_EvalFrameEx + 27144
22 0x7f15c622f6c9p PyEval_EvalCodeEx + 2025
23 0x7f15c61b8567p
24 0x7f15c6193973p PyObject_Call + 67
25 0x7f15c622869ep PyEval_EvalFrameEx + 9486
26 0x7f15c622f6c9p PyEval_EvalCodeEx + 2025
27 0x7f15c622cb98p PyEval_EvalFrameEx + 27144
28 0x7f15c622f6c9p PyEval_EvalCodeEx + 2025
29 0x7f15c622cb98p PyEval_EvalFrameEx + 27144
30 0x7f15c622f6c9p PyEval_EvalCodeEx + 2025
31 0x7f15c622cb98p PyEval_EvalFrameEx + 27144
32 0x7f15c622f6c9p PyEval_EvalCodeEx + 2025
33 0x7f15c622cb98p PyEval_EvalFrameEx + 27144
34 0x7f15c622f6c9p PyEval_EvalCodeEx + 2025
35 0x7f15c61b8567p
36 0x7f15c6193973p PyObject_Call + 67
37 0x7f15c622869ep PyEval_EvalFrameEx + 9486
38 0x7f15c622f6c9p PyEval_EvalCodeEx + 2025
39 0x7f15c61b8567p
40 0x7f15c6193973p PyObject_Call + 67
41 0x7f15c622869ep PyEval_EvalFrameEx + 9486
42 0x7f15c622f6c9p PyEval_EvalCodeEx + 2025
43 0x7f15c622cb98p PyEval_EvalFrameEx + 27144
44 0x7f15c622f6c9p PyEval_EvalCodeEx + 2025
45 0x7f15c622cb98p PyEval_EvalFrameEx + 27144
46 0x7f15c622f6c9p PyEval_EvalCodeEx + 2025
47 0x7f15c61b8567p
48 0x7f15c6193973p PyObject_Call + 67
49 0x7f15c622869ep PyEval_EvalFrameEx + 9486
50 0x7f15c622f6c9p PyEval_EvalCodeEx + 2025
51 0x7f15c622cb98p PyEval_EvalFrameEx + 27144
52 0x7f15c622f6c9p PyEval_EvalCodeEx + 2025
53 0x7f15c622cb98p PyEval_EvalFrameEx + 27144
54 0x7f15c622f6c9p PyEval_EvalCodeEx + 2025
55 0x7f15c622cb98p PyEval_EvalFrameEx + 27144
56 0x7f15c622f6c9p PyEval_EvalCodeEx + 2025
57 0x7f15c622cb98p PyEval_EvalFrameEx + 27144
58 0x7f15c622f6c9p PyEval_EvalCodeEx + 2025
59 0x7f15c622f8eap PyEval_EvalCode + 26
60 0x7f15c622bcc0p PyEval_EvalFrameEx + 23344
61 0x7f15c622f6c9p PyEval_EvalCodeEx + 2025
62 0x7f15c622cb98p PyEval_EvalFrameEx + 27144
63 0x7f15c622f6c9p PyEval_EvalCodeEx + 2025
64 0x7f15c61b847ap
65 0x7f15c6193973p PyObject_Call + 67
66 0x7f15c625c320p
67 0x7f15c625c9dep Py_Main + 1374
68 0x7f15c542d830p __libc_start_main + 240
69 0x555c2751f07fp

0
收藏
回复
全部评论(4)
时间顺序
f
fkcode
#2 回复于2019-10

经高人指点,把克隆那句代码写在优化器前面即可。

0
回复
a
avydhjsv
#3 回复于2019-11

楼主仔细看其中有一句

--> 651 use_program_cache=use_program_cache)

 

我把克隆那句代码写在优化器前面了,但是我训练的时候突然想起来还有代码应该加进去,

然后就终止训练了,然后重新训练的时候,会使用之前的缓存,于是会报错。

建议:关掉重启,再来训练

0
回复
白加黑127
#4 回复于2020-04

但是clone里面不是有for-test=true吗?不就意味着测试时已经默认将优化剪掉了吗?

0
回复
白墙拜拜
#5 回复于2020-05

试过了都不行,还是会出错。。

0
回复
需求/bug反馈?一键提issue告诉我们
发现bug?如果您知道修复办法,欢迎提pr直接参与建设飞桨~
在@后输入用户全名并按空格结束,可艾特全站任一用户