每次读取存盘点并训练完毕后保存存盘点的notebook脚本,如果重复执行会报错如下:
--> 249 self._add_moments_pows(p)
250
251 def _append_optimize_op(self, block, param_and_grad):
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/optimizer/adam.py in _add_moments_pows(self, p)
214 if acc_dtype == core.VarDesc.VarType.FP16:
215 acc_dtype = core.VarDesc.VarType.FP32
--> 216 self._add_accumulator(self._moment1_acc_str, p, dtype=acc_dtype)
217 self._add_accumulator(self._moment2_acc_str, p, dtype=acc_dtype)
218 self._add_accumulator(
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/optimizer/optimizer.py in _add_accumulator(self, name, param, dtype, fill_value, shape, type, device)
532 if len(self._accumulators_holder) > 0:
533 assert var_name in self._accumulators_holder, \
--> 534 "Optimizer set error, {} should in state dict".format( var_name )
535 var.set_value(self._accumulators_holder[var_name])
536
AssertionError: Optimizer set error, linear_0.w_0_moment1_0 should in state dict
如果执行一次后清空缓存再执行就不会报错,这是什么原因?
太难了,我也看不懂啊啊啊
推测可能和显存占用有关吧