W0328 23:10:43.610019 441 gpu_resources.cc:217] WARNING: device: . The installed Paddle is compiled with CUDNN 8.2, but CUDNN version in your machine is 8.1, which may cause serious incompatible bug. Please recompile or reinstall Paddle with compatible CUDNN version.
Traceback (most recent call last):
File "train.py", line 179, in
do_train()
File "train.py", line 142, in do_train
title_token_type_ids=title_token_type_ids)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 948, in __call__
return self.forward(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/parallel.py", line 774, in forward
outputs = self._layers(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 948, in __call__
return self.forward(*inputs, **kwargs)
File "/root/paddlejob/workspace/code/model.py", line 80, in forward
margin_diag = paddle.full(shape=[query_cls_embedding[0]], fill_value=self.margin, dtype='int64')
File "/usr/local/lib/python3.7/dist-packages/paddle/tensor/creation.py", line 897, in full
return fill_constant(shape=shape, dtype=dtype, value=fill_value, name=name)
File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/layers/tensor.py", line 795, in fill_constant
out = _C_ops.full(shape, float(value), dtype, place)
ValueError: (InvalidArgument) Each value of attribute 'shape' is expected to be no less than 0. But recieved: shape[0] = -1; shape = [-1].
[Hint: Expected data[i] >= 0, but received data[i]:-1 < 0:0.] (at /paddle/paddle/phi/infermeta/nullary.cc:38)

LAUNCH INFO 2023-03-28 23:10:46,859 Pod failed
LAUNCH ERROR 2023-03-28 23:10:46,860 Container failed !!!