在张量流训练中更新Keras BatchNormalization总体参数

问题描述

我在带有Cuda 8.0和cuDNN 6的Ubuntu 16.04中将Keras 2.0.8和Tensorflow 1.3.0一起使用

我在模型中使用了两个BatchNormalization层( keras层)，并使用 tensorflow管道进行了训练.

我在这里面临两个问题-

即使训练 K.learning_phase 设置为 True，训练时
1. BatchNorm层人口参数(均值和方差)也不会更新 .结果，推论完全失败了.我需要一些有关如何在培训步骤之间手动更新这些参数的建议.
2. 第二，使用tensorflow 保存器 op 保存经过训练的模型后，当我尝试加载时，结果将无法转载.似乎权重正在改变.在保存负载操作中，有没有办法使权重保持相同?

解决方案

几周前，我遇到了同样的问题.在内部，keras层可以向模型添加其他更新操作(例如 batchnorm ).因此，您需要显式运行这些其他操作.对于batchnorm，这些 updates 似乎只是一些assign_ops，它们将当前的均值/方差与新值交换.如果您不创建keras模型，则可能会起作用.假设 x 是您要标准化的张量

bn = keras.layers.BatchNormalization()
x = bn(x)

....
sess.run([minimizer_op,bn.updates],K.learning_phase(): 1)

在我的工作流程中，我正在创建一个keras模型(不进行编译)，然后运行以下命令

model = keras.Model(inputs=inputs, outputs=prediction)
sess.run([minimizer_op,model.updates],K.learning_phase(): 1)

其中输入可能类似于

inputs = [keras.layers.Input(tensor=input_variables)]

和 outputs 是张量流张量的列表.该模型似乎自动汇总了输入和输出之间的所有其他 updates 操作.

I am using Keras 2.0.8 with Tensorflow 1.3.0 in Ubuntu 16.04 with Cuda 8.0 and cuDNN 6.

I am using two BatchNormalization layers( keras layers ) in my model and training using tensorflow pipeline.

I am facing two problems here -

1. BatchNorm layer population parameters( mean and variance ) are not being updated while training even after setting K.learning_phase to True. As a result, inference is failing completely. I need some advice on how to update these parameters between training steps manually.
2. Secondly, after saving the trained model using tensorflow saver op, when I try to load it, the results cannot be reproduced. It seems the weights are changing. Is there a way to keep the weights same in save-load operation?

解决方案

I ran into the same problem a few weeks ago. Internally, keras layers can add additional update operations to a model (e.g. batchnorm). So you need to run these additional ops explicitly. For the batchnorm these updates seem to be just some assign_ops which swap the current mean/variance with the new values. If you do not create a keras model this might work; assuming x is a tensor you like to normalize

bn = keras.layers.BatchNormalization()
x = bn(x)

....
sess.run([minimizer_op,bn.updates],K.learning_phase(): 1)

In my workflow, I am creating a keras model (w/o compiling it) and then run the following

model = keras.Model(inputs=inputs, outputs=prediction)
sess.run([minimizer_op,model.updates],K.learning_phase(): 1)

where inputs can be something like

inputs = [keras.layers.Input(tensor=input_variables)]

and outputs is a list of tensorflow tensors. The model seems to aggregate all additional updates operations between inputs and outputs automatically.

这篇关于在张量流训练中更新Keras BatchNormalization总体参数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持！