Exception in thread Thread-10: Traceback (most recent call last): File "/home/jiangcx/.conda/envs/paddlex/lib/python3.9/threading.py", line 980, in _bootstrap_inner self.run() File "/home/jiangcx/.conda/envs/paddlex/lib/python3.9/threading.py", line 917, in run self._target(*self._args, **self._kwargs) File "/home/jiangcx/.conda/envs/paddlex/lib/python3.9/site-packages/paddle/io/dataloader/dataloader_iter.py", line 636, in _thread_loop raise e File "/home/jiangcx/.conda/envs/paddlex/lib/python3.9/site-packages/paddle/io/dataloader/dataloader_iter.py", line 617, in thread_loop array.append(tensor) RuntimeError: (PreconditionNotMet) Tensor holds no memory. Call Tensor::mutable_data firstly. [Hint: holder should not be null.] (at ../paddle/phi/core/dense_tensor_impl.cc:44)
Traceback (most recent call last): File "/home/jiangcx/PycharmProjects/PaddleDetection/tools/train.py", line 209, in main() File "/home/jiangcx/PycharmProjects/PaddleDetection/tools/train.py", line 205, in main run(FLAGS, cfg) File "/home/jiangcx/PycharmProjects/PaddleDetection/tools/train.py", line 158, in run trainer.train(FLAGS.eval) File "/home/jiangcx/PycharmProjects/PaddleDetection/ppdet/engine/trainer.py", line 519, in train for step_id, data in enumerate(self.loader): File "/home/jiangcx/PycharmProjects/PaddleDetection/ppdet/data/reader.py", line 214, in next return next(self.loader) File "/home/jiangcx/.conda/envs/paddlex/lib/python3.9/site-packages/paddle/io/dataloader/dataloader_iter.py", line 826, in next self.reader.read_next_list()[0] SystemError: (Fatal) Blocking queue is killed because the data reader raises an exception. [Hint: Expected killed != true, but received killed_:1 == true:1.] (at ../paddle/fluid/operators/reader/blocking_queue.h:174)
训练确实会中断报错： Exception in thread Thread-10: Traceback (most recent call last): File "/home/jiangcx/.conda/envs/paddlex/lib/python3.9/threading.py", line 980, in _bootstrap_inner self.run() File "/home/jiangcx/.conda/envs/paddlex/lib/python3.9/threading.py", line 917, in run self._target(*self._args, **self._kwargs) File "/home/jiangcx/.conda/envs/paddlex/lib/python3.9/site-packages/paddle/io/dataloader/dataloader_iter.py", line 636, in _thread_loop raise e File "/home/jiangcx/.conda/envs/paddlex/lib/python3.9/site-packages/paddle/io/dataloader/dataloader_iter.py", line 617, in thread_loop array.append(tensor) RuntimeError: (PreconditionNotMet) Tensor holds no memory. Call Tensor::mutable_data firstly. [Hint: holder should not be null.] (at ../paddle/phi/core/dense_tensor_impl.cc:44) Traceback (most recent call last): File "/home/jiangcx/PycharmProjects/PaddleDetection/tools/train.py", line 209, in main() File "/home/jiangcx/PycharmProjects/PaddleDetection/tools/train.py", line 205, in main run(FLAGS, cfg) File "/home/jiangcx/PycharmProjects/PaddleDetection/tools/train.py", line 158, in run trainer.train(FLAGS.eval) File "/home/jiangcx/PycharmProjects/PaddleDetection/ppdet/engine/trainer.py", line 519, in train for step_id, data in enumerate(self.loader): File "/home/jiangcx/PycharmProjects/PaddleDetection/ppdet/data/reader.py", line 214, in next return next(self.loader) File "/home/jiangcx/.conda/envs/paddlex/lib/python3.9/site-packages/paddle/io/dataloader/dataloader_iter.py", line 826, in next self.reader.read_next_list()[0] SystemError: (Fatal) Blocking queue is killed because the data reader raises an exception. [Hint: Expected killed != true, but received killed_:1 == true:1.] (at ../paddle/fluid/operators/reader/blocking_queue.h:174)
训练确实会中断报错： Exception in thread Thread-10: Traceback (most recent call last): File "/home/jiangcx/.conda/envs/paddlex/lib/python3.9/threading.py", line 980, in _bootstrap_inner self.run() File "/home/jiangcx/.conda/envs/paddlex/lib/python3.9/threading.py", line 917, in run self._target(*self._args, **self._kwargs) File "/home/jiangcx/.conda/envs/paddlex/lib/python3.9/site-packages/paddle/io/dataloader/dataloader_iter.py", line 636, in _thread_loop raise e File "/home/jiangcx/.conda/envs/paddlex/lib/python3.9/site-packages/paddle/io/dataloader/dataloader_iter.py", line 617, in thread_loop array.append(tensor) RuntimeError: (PreconditionNotMet) Tensor holds no memory. Call Tensor::mutable_data firstly. [Hint: holder should not be null.] (at ../paddle/phi/core/dense_tensor_impl.cc:44) Traceback (most recent call last): File "/home/jiangcx/PycharmProjects/PaddleDetection/tools/train.py", line 209, in main() File "/home/jiangcx/PycharmProjects/PaddleDetection/tools/train.py", line 205, in main run(FLAGS, cfg) File "/home/jiangcx/PycharmProjects/PaddleDetection/tools/train.py", line 158, in run trainer.train(FLAGS.eval) File "/home/jiangcx/PycharmProjects/PaddleDetection/ppdet/engine/trainer.py", line 519, in train for step_id, data in enumerate(self.loader): File "/home/jiangcx/PycharmProjects/PaddleDetection/ppdet/data/reader.py", line 214, in next return next(self.loader) File "/home/jiangcx/.conda/envs/paddlex/lib/python3.9/site-packages/paddle/io/dataloader/dataloader_iter.py", line 826, in next self.reader.read_next_list()[0] SystemError: (Fatal) Blocking queue is killed because the data reader raises an exception. [Hint: Expected killed != true, but received killed_:1 == true:1.] (at ../paddle/fluid/operators/reader/blocking_queue.h:174)
还不是很熟悉Paddle的配置文件，debug了一下，感觉应该默认配置是只训练有标签的数据，COCO api默认self.allow_empty = False
，位置在./PaddleDetection/ppdet/data/source/coco.py 中的class COCODataSet中，修改allow_empty=True就会都读取进来，不然其实在dataset基本就过滤掉了空标签的背景图，这个在外部是否有接口暂时还有点没搞清楚，修改后测试可以实现训练OK背景图的
可以的，感谢
有个问题就是，这里改为True后，训练总会意外中断
训练确实会中断报错：
Exception in thread Thread-10:
Traceback (most recent call last): File "/home/jiangcx/.conda/envs/paddlex/lib/python3.9/threading.py", line 980, in _bootstrap_inner self.run() File "/home/jiangcx/.conda/envs/paddlex/lib/python3.9/threading.py", line 917, in run self._target(*self._args, **self._kwargs) File "/home/jiangcx/.conda/envs/paddlex/lib/python3.9/site-packages/paddle/io/dataloader/dataloader_iter.py", line 636, in _thread_loop raise e File "/home/jiangcx/.conda/envs/paddlex/lib/python3.9/site-packages/paddle/io/dataloader/dataloader_iter.py", line 617, in thread_loop array.append(tensor) RuntimeError: (PreconditionNotMet) Tensor holds no memory. Call Tensor::mutable_data firstly. [Hint: holder should not be null.] (at ../paddle/phi/core/dense_tensor_impl.cc:44)
Traceback (most recent call last): File "/home/jiangcx/PycharmProjects/PaddleDetection/tools/train.py", line 209, in main() File "/home/jiangcx/PycharmProjects/PaddleDetection/tools/train.py", line 205, in main run(FLAGS, cfg) File "/home/jiangcx/PycharmProjects/PaddleDetection/tools/train.py", line 158, in run trainer.train(FLAGS.eval) File "/home/jiangcx/PycharmProjects/PaddleDetection/ppdet/engine/trainer.py", line 519, in train for step_id, data in enumerate(self.loader): File "/home/jiangcx/PycharmProjects/PaddleDetection/ppdet/data/reader.py", line 214, in next return next(self.loader) File "/home/jiangcx/.conda/envs/paddlex/lib/python3.9/site-packages/paddle/io/dataloader/dataloader_iter.py", line 826, in next self.reader.read_next_list()[0] SystemError: (Fatal) Blocking queue is killed because the data reader raises an exception. [Hint: Expected killed != true, but received killed_:1 == true:1.] (at ../paddle/fluid/operators/reader/blocking_queue.h:174)
请问现在这个问题解决了嘛
请问现在这个问题解决了嘛
位置在./PaddleDetection/ppdet/data/source/coco.py中设置了allow_empty=true之后，还需要修改empty_ratio为2，.就默认把所有的背景图放进去，训练的时候显示加载有效图片那个不会改变，因为背景图在代码里不算有效图片，这样就可以训练背景图了
位置在./PaddleDetection/ppdet/data/source/coco.py中设置了allow_empty=true之后，还需要修改empty_ratio为2，.就默认把所有的背景图放进去，训练的时候显示加载有效图片那个不会改变，因为背景图在代码里不算有效图片，这样就可以训练背景图了