首页 Paddle框架 帖子详情
每次训练到最后一轮就会抛出异常,是什么原因?
收藏
快速回复
Paddle框架 问答深度学习模型训练 1096 3
每次训练到最后一轮就会抛出异常,是什么原因?
收藏
快速回复
Paddle框架 问答深度学习模型训练 1096 3

为什么训练每次到最后一轮就会抛出如下异常信息,什么问题导致?

2020-08-01 19:22:44,224-INFO: Best test box ap: 0.8573738037253215, in iter: 3348
FATAL: exception not rethrown
W0801 19:22:44.496243 2730 init.cc:216] Warning: PaddlePaddle catches a failure signal, it may not work properly
W0801 19:22:44.496279 2730 init.cc:218] You could check whether you killed PaddlePaddle thread/process accidentally or report the case to PaddlePaddle
W0801 19:22:44.496285 2730 init.cc:221] The detail failure signal is:

W0801 19:22:44.496294 2730 init.cc:224] *** Aborted at 1596280964 (unix time) try "date -d @1596280964" if you are using GNU date ***
W0801 19:22:44.497895 2730 init.cc:224] PC: @ 0x0 (unknown)
W0801 19:22:44.498033 2730 init.cc:224] *** SIGABRT (@0x3e800000a35) received by PID 2613 (TID 0x7f8ac57fe700) from PID 2613; stack trace: ***
W0801 19:22:44.499296 2730 init.cc:224] @ 0x7f8cf53f9390 (unknown)
W0801 19:22:44.500511 2730 init.cc:224] @ 0x7f8cf5053428 gsignal
W0801 19:22:44.501648 2730 init.cc:224] @ 0x7f8cf505502a abort
W0801 19:22:44.502766 2730 init.cc:224] @ 0x7f8cf50957ea (unknown)
W0801 19:22:44.503866 2730 init.cc:224] @ 0x7f8cf509580e __libc_fatal
W0801 19:22:44.504973 2730 init.cc:224] @ 0x7f8cf53f8030 unwind_cleanup
W0801 19:22:44.505878 2730 init.cc:224] @ 0x7f8ce81fb605 (unknown)
W0801 19:22:44.506880 2730 init.cc:224] @ 0x7f8ce825829a (unknown)
W0801 19:22:44.507768 2730 init.cc:224] @ 0x7f8ce8154c21 (unknown)
W0801 19:22:44.508692 2730 init.cc:224] @ 0x7f8ce7e75585 (unknown)
W0801 19:22:44.509582 2730 init.cc:224] @ 0x7f8ce7fec2a4 (unknown)
W0801 19:22:44.509883 2730 init.cc:224] @ 0x56532f0a9744 _PyMethodDef_RawFastCallKeywords
W0801 19:22:44.510138 2730 init.cc:224] @ 0x56532f0a9861 _PyCFunction_FastCallKeywords
W0801 19:22:44.510393 2730 init.cc:224] @ 0x56532f1152bd _PyEval_EvalFrameDefault
W0801 19:22:44.510617 2730 init.cc:224] @ 0x56532f059539 _PyEval_EvalCodeWithName
W0801 19:22:44.510834 2730 init.cc:224] @ 0x56532f05a635 _PyFunction_FastCallDict
W0801 19:22:44.511054 2730 init.cc:224] @ 0x56532f078e53 _PyObject_Call_Prepend
W0801 19:22:44.511188 2730 init.cc:224] @ 0x56532f0b0a3a slot_tp_call
W0801 19:22:44.511436 2730 init.cc:224] @ 0x56532f0b18fb _PyObject_FastCallKeywords
W0801 19:22:44.511698 2730 init.cc:224] @ 0x56532f114e86 _PyEval_EvalFrameDefault
W0801 19:22:44.511968 2730 init.cc:224] @ 0x56532f05a56b _PyFunction_FastCallDict
W0801 19:22:44.512212 2730 init.cc:224] @ 0x56532f078e53 _PyObject_Call_Prepend
W0801 19:22:44.512341 2730 init.cc:224] @ 0x56532f0b0a3a slot_tp_call
W0801 19:22:44.512574 2730 init.cc:224] @ 0x56532f0b18fb _PyObject_FastCallKeywords
W0801 19:22:44.512835 2730 init.cc:224] @ 0x56532f1156e8 _PyEval_EvalFrameDefault
W0801 19:22:44.513047 2730 init.cc:224] @ 0x56532f059539 _PyEval_EvalCodeWithName
W0801 19:22:44.513273 2730 init.cc:224] @ 0x56532f05a635 _PyFunction_FastCallDict
W0801 19:22:44.513489 2730 init.cc:224] @ 0x56532f078e53 _PyObject_Call_Prepend
W0801 19:22:44.513727 2730 init.cc:224] @ 0x56532f06bdbe PyObject_Call
W0801 19:22:44.513878 2730 init.cc:224] @ 0x56532f168bc7 partial_call
W0801 19:22:44.514117 2730 init.cc:224] @ 0x56532f0b18fb _PyObject_FastCallKeywords
W0801 19:22:44.514360 2730 init.cc:224] @ 0x56532f114e86 _PyEval_EvalFrameDefault
Aborted (core dumped)

0
收藏
回复
全部评论(3)
时间顺序
尹亦宇
#2 回复于2020-08

可以把代码或者地址发出来看一看,当然也有可能是网络问题

0
回复
lianzhang132
#3 回复于2020-08

感觉是内存问题   把 batch调小点试试

0
回复
Mr.郑先生_
#4 回复于2020-08

先跑一轮试试看有没有问题,我感觉是代码的问题

0
回复
需求/bug反馈?一键提issue告诉我们
发现bug?如果您知道修复办法,欢迎提pr直接参与建设飞桨~
在@后输入用户全名并按空格结束,可艾特全站任一用户