Batch Norm#
Batch Norm layers normalize the activations of the previous layer such that the mean activation is close to 0 and the standard deviation is close to 1. Adding Batch Norm reduces the amount of covariate shift within the network which makes it possible to use higher learning rates and thus converge faster under some circumstances.
Parameters#
# | Name | Default | Type | Description |
---|---|---|---|---|
1 | decay | 0.9 | float | The decay rate of the previous running averages of the global mean and variance. |
2 | betaInitializer | Constant | Initializer | The initializer of the beta parameter. |
3 | gammaInitializer | Constant | Initializer | The initializer of the gamma parameter. |
Example#
use Rubix\ML\NeuralNet\Layers\BatchNorm;
use Rubix\ML\NeuralNet\Initializers\Constant;
use Rubix\ML\NeuralNet\Initializers\Normal;
$layer = new BatchNorm(0.7, new Constant(0.), new Normal(1.));
References#
-
S. Ioffe et al. (2015). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. ↩