Paddle的BatchNorm在组网时beta，gamma参数不能训练？
收藏
我用Keras和Paddle搭建了同一个网络，他们的 Total params ：556,383
但是 Keras 的 Non-trainable params 是1,792
而Paddle搭建的Non-trainable params 是3,584
我查看了网络结构，发现归一化层的参数被当作非训练参数了，我该如何调整？
我查阅了Paddle的文档，发现并没有对beta 和 gamma的叙述
Paddle的网络框架
---------------------------------------------------------------------------
Layer (type) Input Shape Output Shape Param #
===========================================================================
Conv2D-64 [[1, 31, 32, 32]] [1, 64, 32, 32] 17,920
ReLU-60 [[1, 64, 32, 32]] [1, 64, 32, 32] 0
Conv2D-50 [[1, 64, 32, 32]] [1, 64, 32, 32] 36,928
BatchNorm-43 [[1, 64, 32, 32]] [1, 64, 32, 32] 256
ReLU-46 [[1, 64, 32, 32]] [1, 64, 32, 32] 0
DHSIS_block-44 [[1, 64, 32, 32]] [1, 64, 32, 32] 0
Conv2D-51 [[1, 64, 32, 32]] [1, 64, 32, 32] 36,928
BatchNorm-44 [[1, 64, 32, 32]] [1, 64, 32, 32] 256
ReLU-47 [[1, 64, 32, 32]] [1, 64, 32, 32] 0
DHSIS_block-45 [[1, 64, 32, 32]] [1, 64, 32, 32] 0
Conv2D-52 [[1, 64, 32, 32]] [1, 64, 32, 32] 36,928
BatchNorm-45 [[1, 64, 32, 32]] [1, 64, 32, 32] 256
ReLU-48 [[1, 64, 32, 32]] [1, 64, 32, 32] 0
DHSIS_block-46 [[1, 64, 32, 32]] [1, 64, 32, 32] 0
Conv2D-53 [[1, 64, 32, 32]] [1, 64, 32, 32] 36,928
BatchNorm-46 [[1, 64, 32, 32]] [1, 64, 32, 32] 256
ReLU-49 [[1, 64, 32, 32]] [1, 64, 32, 32] 0
DHSIS_block-47 [[1, 64, 32, 32]] [1, 64, 32, 32] 0
Conv2D-54 [[1, 64, 32, 32]] [1, 64, 32, 32] 36,928
BatchNorm-47 [[1, 64, 32, 32]] [1, 64, 32, 32] 256
ReLU-50 [[1, 64, 32, 32]] [1, 64, 32, 32] 0
DHSIS_block-48 [[1, 64, 32, 32]] [1, 64, 32, 32] 0
Conv2D-55 [[1, 64, 32, 32]] [1, 64, 32, 32] 36,928
BatchNorm-48 [[1, 64, 32, 32]] [1, 64, 32, 32] 256
ReLU-51 [[1, 64, 32, 32]] [1, 64, 32, 32] 0
DHSIS_block-49 [[1, 64, 32, 32]] [1, 64, 32, 32] 0
Conv2D-56 [[1, 64, 32, 32]] [1, 64, 32, 32] 36,928
BatchNorm-49 [[1, 64, 32, 32]] [1, 64, 32, 32] 256
ReLU-52 [[1, 64, 32, 32]] [1, 64, 32, 32] 0
DHSIS_block-50 [[1, 64, 32, 32]] [1, 64, 32, 32] 0
Conv2D-57 [[1, 64, 32, 32]] [1, 64, 32, 32] 36,928
BatchNorm-50 [[1, 64, 32, 32]] [1, 64, 32, 32] 256
ReLU-53 [[1, 64, 32, 32]] [1, 64, 32, 32] 0
DHSIS_block-51 [[1, 64, 32, 32]] [1, 64, 32, 32] 0
Conv2D-58 [[1, 64, 32, 32]] [1, 64, 32, 32] 36,928
BatchNorm-51 [[1, 64, 32, 32]] [1, 64, 32, 32] 256
ReLU-54 [[1, 64, 32, 32]] [1, 64, 32, 32] 0
DHSIS_block-52 [[1, 64, 32, 32]] [1, 64, 32, 32] 0
Conv2D-59 [[1, 64, 32, 32]] [1, 64, 32, 32] 36,928
BatchNorm-52 [[1, 64, 32, 32]] [1, 64, 32, 32] 256
ReLU-55 [[1, 64, 32, 32]] [1, 64, 32, 32] 0
DHSIS_block-53 [[1, 64, 32, 32]] [1, 64, 32, 32] 0
Conv2D-60 [[1, 64, 32, 32]] [1, 64, 32, 32] 36,928
BatchNorm-53 [[1, 64, 32, 32]] [1, 64, 32, 32] 256
ReLU-56 [[1, 64, 32, 32]] [1, 64, 32, 32] 0
DHSIS_block-54 [[1, 64, 32, 32]] [1, 64, 32, 32] 0
Conv2D-61 [[1, 64, 32, 32]] [1, 64, 32, 32] 36,928
BatchNorm-54 [[1, 64, 32, 32]] [1, 64, 32, 32] 256
ReLU-57 [[1, 64, 32, 32]] [1, 64, 32, 32] 0
DHSIS_block-55 [[1, 64, 32, 32]] [1, 64, 32, 32] 0
Conv2D-62 [[1, 64, 32, 32]] [1, 64, 32, 32] 36,928
BatchNorm-55 [[1, 64, 32, 32]] [1, 64, 32, 32] 256
ReLU-58 [[1, 64, 32, 32]] [1, 64, 32, 32] 0
DHSIS_block-56 [[1, 64, 32, 32]] [1, 64, 32, 32] 0
Conv2D-63 [[1, 64, 32, 32]] [1, 64, 32, 32] 36,928
BatchNorm-56 [[1, 64, 32, 32]] [1, 64, 32, 32] 256
ReLU-59 [[1, 64, 32, 32]] [1, 64, 32, 32] 0
DHSIS_block-57 [[1, 64, 32, 32]] [1, 64, 32, 32] 0
Conv2D-65 [[1, 64, 32, 32]] [1, 31, 32, 32] 17,887
===========================================================================
Total params: 556,383
Trainable params: 552,799
Non-trainable params: 3,584
---------------------------------------------------------------------------
Input size (MB): 0.12
Forward/backward pass size (MB): 29.24
Params size (MB): 2.12
Estimated Total Size (MB): 31.49
---------------------------------------------------------------------------
Keras（tf 1.5.0） 的网络框架：
_________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input (InputLayer) (None, 32, 32, 31) 0 _________________________________________________________________ conv11 (Conv2D) (None, 32, 32, 64) 17920 _________________________________________________________________ conv11_relu (Activation) (None, 32, 32, 64) 0 _________________________________________________________________ conv12 (Conv2D) (None, 32, 32, 64) 36928 _________________________________________________________________ conv12_bn (BatchNormalizatio (None, 32, 32, 64) 256 _________________________________________________________________ conv12_relu (Activation) (None, 32, 32, 64) 0 _________________________________________________________________ conv13 (Conv2D) (None, 32, 32, 64) 36928 _________________________________________________________________ conv13_bn (BatchNormalizatio (None, 32, 32, 64) 256 _________________________________________________________________ conv13_relu (Activation) (None, 32, 32, 64) 0 _________________________________________________________________ conv14 (Conv2D) (None, 32, 32, 64) 36928 _________________________________________________________________ conv14_bn (BatchNormalizatio (None, 32, 32, 64) 256 _________________________________________________________________ conv14relu (Activation) (None, 32, 32, 64) 0 _________________________________________________________________ conv15 (Conv2D) (None, 32, 32, 64) 36928 _________________________________________________________________ conv15_bn (BatchNormalizatio (None, 32, 32, 64) 256 _________________________________________________________________ conv15_relu (Activation) (None, 32, 32, 64) 0 _________________________________________________________________ conv16 (Conv2D) (None, 32, 32, 64) 36928 _________________________________________________________________ conv16_bn (BatchNormalizatio (None, 32, 32, 64) 256 _________________________________________________________________ conv16_relu (Activation) (None, 32, 32, 64) 0 _________________________________________________________________ conv17 (Conv2D) (None, 32, 32, 64) 36928 _________________________________________________________________ conv17_bn (BatchNormalizatio (None, 32, 32, 64) 256 _________________________________________________________________ conv17_relu (Activation) (None, 32, 32, 64) 0 _________________________________________________________________ conv18 (Conv2D) (None, 32, 32, 64) 36928 _________________________________________________________________ conv18_bn (BatchNormalizatio (None, 32, 32, 64) 256 _________________________________________________________________ conv18_relu (Activation) (None, 32, 32, 64) 0 _________________________________________________________________ conv19 (Conv2D) (None, 32, 32, 64) 36928 _________________________________________________________________ conv19_bn (BatchNormalizatio (None, 32, 32, 64) 256 _________________________________________________________________ conv19_relu (Activation) (None, 32, 32, 64) 0 _________________________________________________________________ conv20 (Conv2D) (None, 32, 32, 64) 36928 _________________________________________________________________ conv20_bn (BatchNormalizatio (None, 32, 32, 64) 256 _________________________________________________________________ conv20_relu (Activation) (None, 32, 32, 64) 0 _________________________________________________________________ conv21 (Conv2D) (None, 32, 32, 64) 36928 _________________________________________________________________ conv21_bn (BatchNormalizatio (None, 32, 32, 64) 256 _________________________________________________________________ conv21_relu (Activation) (None, 32, 32, 64) 0 _________________________________________________________________ conv22 (Conv2D) (None, 32, 32, 64) 36928 _________________________________________________________________ conv22_bn (BatchNormalizatio (None, 32, 32, 64) 256 _________________________________________________________________ conv22_relu (Activation) (None, 32, 32, 64) 0 _________________________________________________________________ conv23 (Conv2D) (None, 32, 32, 64) 36928 _________________________________________________________________ conv23_bn (BatchNormalizatio (None, 32, 32, 64) 256 _________________________________________________________________ conv23_relu (Activation) (None, 32, 32, 64) 0 _________________________________________________________________ conv24 (Conv2D) (None, 32, 32, 64) 36928 _________________________________________________________________ conv24_bn (BatchNormalizatio (None, 32, 32, 64) 256 _________________________________________________________________ conv24_relu (Activation) (None, 32, 32, 64) 0 _________________________________________________________________ conv25 (Conv2D) (None, 32, 32, 64) 36928 _________________________________________________________________ conv25_bn (BatchNormalizatio (None, 32, 32, 64) 256 _________________________________________________________________ conv25_relu (Activation) (None, 32, 32, 64) 0 _________________________________________________________________ conv30 (Conv2D) (None, 32, 32, 31) 17887 ================================================================= Total params: 556,383 Trainable params: 554,591 Non-trainable params: 1,792 _________________________________________________________________
XiangLiu
已解决
5#
回复于2022-06
正常使用nn.BatchNorm2D(num_channels)，在net.train()模式下beat、gamma是训练的，mean，variance也是更新的。在net.eval()模式下beta、gamma不再更新，mean、variance使用训练阶段获得的最终均值方差。
0
收藏
请登录后评论
请问疑惑解决了吗，我也发现了这个问题
没解决
经过测试，只是显示参数不同，实际计算结果相同。
正常使用nn.BatchNorm2D(num_channels)，在net.train()模式下beat、gamma是训练的，mean，variance也是更新的。在net.eval()模式下beta、gamma不再更新，mean、variance使用训练阶段获得的最终均值方差。
长知识了