首页 Paddle框架 帖子详情
训练mask_rcnn出现错误,请问什么原因?
收藏
快速回复
Paddle框架 问答深度学习模型训练 1705 2
训练mask_rcnn出现错误,请问什么原因?
收藏
快速回复
Paddle框架 问答深度学习模型训练 1705 2

paddledetection 训练自定义COCO数据集出现错误:

2020-07-11 19:01:24,575-INFO: iter: 40, lr: 0.003867, 'loss_cls': '0.149100', 'loss_bbox': '0.052158', 'loss_rpn_cls': '0.449343', 'loss_rpn_bbox': '0.152205', 'loss_mask': '0.522817', 'loss': '1.363394', time: 4.511, eta: 18 days, 19:01:36
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/numpy/lib/function_base.py:3405: RuntimeWarning: Invalid value encountered in median
r = func(a, **kwargs)
W0711 19:02:05.036720 204 operator.cc:187] generate_mask_labels raises an exception std::out_of_range, vector::_M_range_check
F0711 19:02:05.036840 204 exception_holder.h:37] std::exception caught, vector::_M_range_check
*** Check failure stack trace: ***
@ 0x7f9803d12a3d google::LogMessage::Fail()
@ 0x7f9803d164ec google::LogMessage::SendToLog()
W0711 19:02:05.038872 205 operator.cc:187] generate_mask_labels raises an exception std::out_of_range, vector::_M_range_check
F0711 19:02:05.038938 205 exception_holder.h:37] std::exception caught, vector::_M_range_check
*** Check failure stack trace: ***
@ 0x7f9803d12a3d google::LogMessage::Fail()
@ 0x7f9803d12563 google::LogMessage::Flush()
@ 0x7f9803d164ec google::LogMessage::SendToLog()
@ 0x7f9803d179fe google::LogMessageFatal::~LogMessageFatal()
@ 0x7f98057d9e18 paddle::framework::details::ExceptionHolder::Catch()
@ 0x7f9803d12563 google::LogMessage::Flush()
@ 0x7f980585cc1e paddle::framework::details::FastThreadedSSAGraphExecutor::RunOpSync()
@ 0x7f9803d179fe google::LogMessageFatal::~LogMessageFatal()
@ 0x7f98057d9e18 paddle::framework::details::ExceptionHolder::Catch()
@ 0x7f980585a5bf paddle::framework::details::FastThreadedSSAGraphExecutor::RunOp()
@ 0x7f980585a884 _ZNSt17_Function_handlerIFvvESt17reference_wrapperISt12_Bind_simpleIFS1_ISt5_BindIFZN6paddle9framework7details28FastThreadedSSAGraphExecutor10RunOpAsyncEPSt13unordered_mapIPNS6_12OpHandleBaseESt6atomicIiESt4hashISA_ESt8equal_toISA_ESaISt4pairIKSA_SC_EEESA_RKSt10shared_ptrINS5_13BlockingQueueImEEEEUlvE_vEEEvEEEE9_M_invokeERKSt9_Any_data
@ 0x7f980585cc1e paddle::framework::details::FastThreadedSSAGraphExecutor::RunOpSync()
@ 0x7f9803d709f3 std::_Function_handler<>::_M_invoke()
@ 0x7f980585a5bf paddle::framework::details::FastThreadedSSAGraphExecutor::RunOp()
@ 0x7f980585a884 _ZNSt17_Function_handlerIFvvESt17reference_wrapperISt12_Bind_simpleIFS1_ISt5_BindIFZN6paddle9framework7details28FastThreadedSSAGraphExecutor10RunOpAsyncEPSt13unordered_mapIPNS6_12OpHandleBaseESt6atomicIiESt4hashISA_ESt8equal_toISA_ESaISt4pairIKSA_SC_EEESA_RKSt10shared_ptrINS5_13BlockingQueueImEEEEUlvE_vEEEvEEEE9_M_invokeERKSt9_Any_data
@ 0x7f9803d709f3 std::_Function_handler<>::_M_invoke()
@ 0x7f9803b7b4f7 std::__future_base::_State_base::_M_do_set()
@ 0x7f9803b7b4f7 std::__future_base::_State_base::_M_do_set()
@ 0x7f9823714a99 __pthread_once_slow
@ 0x7f9823714a99 __pthread_once_slow
@ 0x7f9805856a52 _ZNSt13__future_base11_Task_stateISt5_BindIFZN6paddle9framework7details28FastThreadedSSAGraphExecutor10RunOpAsyncEPSt13unordered_mapIPNS4_12OpHandleBaseESt6atomicIiESt4hashIS8_ESt8equal_toIS8_ESaISt4pairIKS8_SA_EEES8_RKSt10shared_ptrINS3_13BlockingQueueImEEEEUlvE_vEESaIiEFvvEE6_M_runEv
@ 0x7f9805856a52 _ZNSt13__future_base11_Task_stateISt5_BindIFZN6paddle9framework7details28FastThreadedSSAGraphExecutor10RunOpAsyncEPSt13unordered_mapIPNS4_12OpHandleBaseESt6atomicIiESt4hashIS8_ESt8equal_toIS8_ESaISt4pairIKS8_SA_EEES8_RKSt10shared_ptrINS3_13BlockingQueueImEEEEUlvE_vEESaIiEFvvEE6_M_runEv
@ 0x7f9803b7d954 _ZZN10ThreadPoolC1EmENKUlvE_clEv
@ 0x7f9803b7d954 _ZZN10ThreadPoolC1EmENKUlvE_clEv
@ 0x7f981d30f421 execute_native_thread_routine_compat
@ 0x7f981d30f421 execute_native_thread_routine_compat
@ 0x7f982370d6ba start_thread
@ 0x7f982370d6ba start_thread
@ 0x7f982344341d clone
@ 0x7f982344341d clone
@ (nil) (unknown)
@ (nil) (unknown)
Aborted (core dumped)
aistudio@jupyter-332130-624453:~/work/PaddleDetection-release-0.3$

0
收藏
回复
全部评论(2)
时间顺序
HolliZhao
#2 回复于2020-07

训练过程当中出错了,感觉是不是有脏数据啊。 检查下iter: 40的mask_labels文件看看。

0
回复
一只烂笔头
#3 回复于2020-07

谢谢回复。可能存在部分训练图片分比率不一致导致,剔除后,并且改GPU方式训练,解决了该问题。

0
回复
需求/bug反馈?一键提issue告诉我们
发现bug?如果您知道修复办法,欢迎提pr直接参与建设飞桨~
在@后输入用户全名并按空格结束,可艾特全站任一用户