PaddleDetection训练的时候，如果Config文件里定义的类别数少于实际数据集中的类别数，会出现如下报错，训练会卡住

2021-02-08 10:05:18,398-INFO: Decompressing /root/.cache/paddle/weights/ResNet50_vd_ssld_pretrained.tar...
2021-02-08 10:05:19,740-WARNING: /root/.cache/paddle/weights/ResNet50_vd_ssld_pretrained.pdparams not found, try to load model file saved with [ save_params, save_persistables, save_vars ]
/opt/_internal/cpython-3.7.0/lib/python3.7/site-packages/paddle/fluid/io.py:1998: UserWarning: This list is not set, Because of Paramerter not found in program. There are: fc_0.b_0 fc_0.w_0
  format(" ".join(unused_para_list)))
2021-02-08 10:05:31,989-WARNING: fail to map op [Gt2YoloTarget_d6e1e3] with error: index 9 is out of bounds for axis 1 with size 7 and stack:
Traceback (most recent call last):
  File "/root/paddlejob/workspace/code/PaddleDetection/ppdet/data/reader.py", line 46, in __call__
    data = f(data, ctx)
  File "/root/paddlejob/workspace/code/PaddleDetection/ppdet/data/transform/batch_operators.py", line 279, in __call__
    target[best_n, 6 + cls, gj, gi] = 1.
IndexError: index 9 is out of bounds for axis 1 with size 7

2021-02-08 10:05:31,995-WARNING: recv endsignal from outq with errmsg[consumer[consumer-ef7-3] failed to map with error:[index 9 is out of bounds for axis 1 with size 7]]
2021-02-08 10:05:32,053-WARNING: fail to map op [Gt2YoloTarget_d6e1e3] with error: index 9 is out of bounds for axis 1 with size 7 and stack:
Traceback (most recent call last):
  File "/root/paddlejob/workspace/code/PaddleDetection/ppdet/data/reader.py", line 46, in __call__
    data = f(data, ctx)
  File "/root/paddlejob/workspace/code/PaddleDetection/ppdet/data/transform/batch_operators.py", line 279, in __call__
    target[best_n, 6 + cls, gj, gi] = 1.
IndexError: index 9 is out of bounds for axis 1 with size 7

2021-02-08 10:05:32,094-WARNING: fail to map op [Gt2YoloTarget_d6e1e3] with error: index 9 is out of bounds for axis 1 with size 7 and stack:
Traceback (most recent call last):
  File "/root/paddlejob/workspace/code/PaddleDetection/ppdet/data/reader.py", line 46, in __call__
    data = f(data, ctx)
  File "/root/paddlejob/workspace/code/PaddleDetection/ppdet/data/transform/batch_operators.py", line 279, in __call__
    target[best_n, 6 + cls, gj, gi] = 1.
IndexError: index 9 is out of bounds for axis 1 with size 7

2021-02-08 10:05:32,115-WARNING: fail to map op [Gt2YoloTarget_d6e1e3] with error: index 9 is out of bounds for axis 1 with size 7 and stack:
Traceback (most recent call last):
  File "/root/paddlejob/workspace/code/PaddleDetection/ppdet/data/reader.py", line 46, in __call__
    data = f(data, ctx)
  File "/root/paddlejob/workspace/code/PaddleDetection/ppdet/data/transform/batch_operators.py", line 279, in __call__
    target[best_n, 6 + cls, gj, gi] = 1.
IndexError: index 9 is out of bounds for axis 1 with size 7

2021-02-08 10:05:32,122-WARNING: fail to map op [Gt2YoloTarget_d6e1e3] with error: index 9 is out of bounds for axis 1 with size 7 and stack:
Traceback (most recent call last):
  File "/root/paddlejob/workspace/code/PaddleDetection/ppdet/data/reader.py", line 46, in __call__
    data = f(data, ctx)
  File "/root/paddlejob/workspace/code/PaddleDetection/ppdet/data/transform/batch_operators.py", line 279, in __call__
    target[best_n, 6 + cls, gj, gi] = 1.
IndexError: index 9 is out of bounds for axis 1 with size 7

2021-02-08 10:05:32,138-WARNING: fail to map op [Gt2YoloTarget_d6e1e3] with error: index 9 is out of bounds for axis 1 with size 7 and stack:
Traceback (most recent call last):
  File "/root/paddlejob/workspace/code/PaddleDetection/ppdet/data/reader.py", line 46, in __call__
    data = f(data, ctx)
  File "/root/paddlejob/workspace/code/PaddleDetection/ppdet/data/transform/batch_operators.py", line 279, in __call__
    target[best_n, 6 + cls, gj, gi] = 1.
IndexError: index 9 is out of bounds for axis 1 with size 7

2021-02-08 10:05:32,195-WARNING: fail to map op [Gt2YoloTarget_d6e1e3] with error: index 9 is out of bounds for axis 1 with size 7 and stack:
Traceback (most recent call last):
  File "/root/paddlejob/workspace/code/PaddleDetection/ppdet/data/reader.py", line 46, in __call__
    data = f(data, ctx)
  File "/root/paddlejob/workspace/code/PaddleDetection/ppdet/data/transform/batch_operators.py", line 279, in __call__
    target[best_n, 6 + cls, gj, gi] = 1.
IndexError: index 9 is out of bounds for axis 1 with size 7

2021-02-08 10:05:32,900-WARNING: fail to map op [Gt2YoloTarget_d6e1e3] with error: index 9 is out of bounds for axis 1 with size 7 and stack:
Traceback (most recent call last):
  File "/root/paddlejob/workspace/code/PaddleDetection/ppdet/data/reader.py", line 46, in __call__
    data = f(data, ctx)
  File "/root/paddlejob/workspace/code/PaddleDetection/ppdet/data/transform/batch_operators.py", line 279, in __call__
    target[best_n, 6 + cls, gj, gi] = 1.
IndexError: index 9 is out of bounds for axis 1 with size 7

个人觉得这个地方还是有些改进空间的。

首先，确实类别设置错误的情况还蛮普遍的，但一般是设多了，而不是设少了（比如默认的coco类别数没改过来）

但如果确实遇到这个情况，上面的报错其实并不能特别看明白问题在那，我是反复找了半天，猜到的。

另外一个问题是，报错后，训练不会跳出，会直接卡住，继续占用计算资源，这个设计个人认为还是有些改进空间的。