训练报错check_guards()!=true

项目

数据集

课程

比赛

模型库

活动

论坛

访问飞桨官网

项目

数据集

课程

比赛

模型库

活动

论坛

访问飞桨官网

Hazelnut 发布于2020-02

训练报错：Expected desc->check_guards() == true, but received desc->check_guards():0 != true

环境：AI Studio CPU环境

训练过程打印的完整内容如下：

/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/executor.py:779: UserWarning: The following exception is not an EOF exception.
"The following exception is not an EOF exception.")
Traceback (most recent call last):
File "train.py", line 136, in
cost_val, = exe.run(program=main_program, feed={'data':img, 'label':lab}, fetch_list=[cost])
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/executor.py", line 780, in run
six.reraise(*sys.exc_info())
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/six.py", line 693, in reraise
raise value
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/executor.py", line 775, in run
use_program_cache=use_program_cache)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/executor.py", line 822, in _run_impl
use_program_cache=use_program_cache)
File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/executor.py", line 899, in _run_program
fetch_var_name)
paddle.fluid.core_avx.EnforceNotMet:

--------------------------------------------
C++ Call Stacks (More useful to developers):
--------------------------------------------
0 std::string paddle::platform::GetTraceBackString(std::string const&, char const*, int)
1 paddle::memory::detail::MetadataCache::load(paddle::memory::detail::MemoryBlock const*) const
2 paddle::memory::detail::MemoryBlock::type(paddle::memory::detail::MetadataCache const&) const
3 paddle::memory::detail::BuddyAllocator::Free(void*)
4 void paddle::memory::legacy::Free(paddle::platform::CPUPlace const&, void*, unsigned long)
5 paddle::memory::allocation::NaiveBestFitAllocator::FreeImpl(paddle::memory::allocation::Allocation*)
6 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()
7 std::deque, std::allocator > >::~deque()
8 std::_Function_handler, std::allocator > >, paddle::framework::GarbageCollector::Add, std::allocator > > >(std::deque, std::allocator > >&&)::{lambda()#1}>(std::deque, std::allocator > >&&, paddle::framework::GarbageCollector::Add, std::allocator > > >(std::deque, std::allocator > >&&)::{lambda()#1}&&)::{lambda()#1}>::_M_invoke(std::_Any_data const&)
9 void paddle::framework::GarbageCollector::Add, std::allocator > >, paddle::framework::GarbageCollector::Add, std::allocator > > >(std::deque, std::allocator > >&&)::{lambda()#1}>(std::deque, std::allocator > >&&, paddle::framework::GarbageCollector::Add, std::allocator > > >(std::deque, std::allocator > >&&)::{lambda()#1}&&)
10 paddle::framework::DeleteUnusedTensors(paddle::framework::Scope const&, paddle::framework::OperatorBase const*, std::unordered_map >, std::hash, std::equal_to, std::allocator > > > > const&, paddle::framework::GarbageCollector*)
11 paddle::framework::Executor::RunPreparedContext(paddle::framework::ExecutorPrepareContext*, paddle::framework::Scope*, bool, bool, bool)
12 paddle::framework::Executor::Run(paddle::framework::ProgramDesc const&, paddle::framework::Scope*, int, bool, bool, std::vector > const&, bool)

----------------------
Error Message Summary:
----------------------
Error: Paddle internal Check failed. (Please help us create a new issue, here we need to find the developer to add a user friendly error message)
[Hint: Expected desc->check_guards() == true, but received desc->check_guards():0 != true:1.] at (/paddle/paddle/fluid/memory/detail/meta_cache.cc:33)

terminate called after throwing an instance of 'paddle::platform::EnforceNotMet'
what():

W0205 19:15:52.376451 480 init.cc:206] *** Aborted at 1580901352 (unix time) try "date -d @1580901352" if you are using GNU date ***
W0205 19:15:52.377750 480 init.cc:206] PC: @ 0x0 (unknown)
W0205 19:15:52.377836 480 init.cc:206] *** SIGABRT (@0x3e8000001e0) received by PID 480 (TID 0x7f9ee0a9f700) from PID 480; stack trace: ***
W0205 19:15:52.378739 480 init.cc:206] @ 0x7f9ee067c390 (unknown)
W0205 19:15:52.379601 480 init.cc:206] @ 0x7f9ee02d6428 gsignal
W0205 19:15:52.380453 480 init.cc:206] @ 0x7f9ee02d802a abort
W0205 19:15:52.381194 480 init.cc:206] @ 0x7f9eda44a84a __gnu_cxx::__verbose_terminate_handler()
W0205 19:15:52.381819 480 init.cc:206] @ 0x7f9eda448f47 __cxxabiv1::__terminate()
W0205 19:15:52.382416 480 init.cc:206] @ 0x7f9eda4483a5 __cxa_call_terminate
W0205 19:15:52.383056 480 init.cc:206] @ 0x7f9eda448bd8 __gxx_personality_v0
W0205 19:15:52.383716 480 init.cc:206] @ 0x7f9edbe29aab _Unwind_RaiseException_Phase2
W0205 19:15:52.384366 480 init.cc:206] @ 0x7f9edbe29f49 _Unwind_Resume
W0205 19:15:52.384915 480 init.cc:206] @ 0x7f9ec2fa123d paddle::memory::detail::BuddyAllocator::Free()
W0205 19:15:52.386375 480 init.cc:206] @ 0x7f9ec2f93ee5 paddle::memory::legacy::Free<>()
W0205 19:15:52.387081 480 init.cc:206] @ 0x7f9ec2f94605 paddle::memory::allocation::NaiveBestFitAllocator::FreeImpl()
W0205 19:15:52.387548 480 init.cc:206] @ 0x7f9ec1754d19 std::_Sp_counted_base<>::_M_release()
W0205 19:15:52.388042 480 init.cc:206] @ 0x7f9ec17557a8 paddle::framework::Variable::PlaceholderImpl<>::~PlaceholderImpl()
W0205 19:15:52.388514 480 init.cc:206] @ 0x7f9ec2f6dd0d paddle::framework::Scope::~Scope()
W0205 19:15:52.389405 480 init.cc:206] @ 0x7f9ec2f6dc21 paddle::framework::Scope::DropKids()
W0205 19:15:52.389900 480 init.cc:206] @ 0x7f9ec2f6dc8d paddle::framework::Scope::~Scope()
W0205 19:15:52.390259 480 init.cc:206] @ 0x7f9ec19769a6 paddle::framework::ScopePool::DeleteScope()
W0205 19:15:52.390856 480 init.cc:206] @ 0x7f9ec1976a01 paddle::framework::ScopePool::Clear()
W0205 19:15:52.390931 480 init.cc:206] @ 0x55d8a6fff5a3 capsule_dealloc.cold.387
W0205 19:15:52.391136 480 init.cc:206] @ 0x55d8a70a1651 PyDict_Clear
W0205 19:15:52.391191 480 init.cc:206] @ 0x55d8a70a171a dict_tp_clear
W0205 19:15:52.391254 480 init.cc:206] @ 0x55d8a70c7648 collect
W0205 19:15:52.391433 480 init.cc:206] @ 0x55d8a71bffda _PyGC_CollectNoFail
W0205 19:15:52.391607 480 init.cc:206] @ 0x55d8a714bbcc PyImport_Cleanup
W0205 19:15:52.391801 480 init.cc:206] @ 0x55d8a71c7087 Py_FinalizeEx
W0205 19:15:52.391934 480 init.cc:206] @ 0x55d8a71d9f93 pymain_main
W0205 19:15:52.392103 480 init.cc:206] @ 0x55d8a71da2bc _Py_UnixMain
W0205 19:15:52.393163 480 init.cc:206] @ 0x7f9ee02c1830 __libc_start_main
W0205 19:15:52.393512 480 init.cc:206] @ 0x55d8a717f062 (unknown)
Aborted (core dumped)

提issue

需求/bug反馈？一键提issue告诉我们

提pr

发现bug？如果您知道修复办法，欢迎提pr直接参与建设飞桨~