The Cyclical optimizer uses a global learning rate that cycles between the lower and upper bound over a designated period while also decaying the upper bound by a factor at each step. Cyclical learning rates have been shown to help escape bad local minima and saddle points of the gradient.
|1||lower||0.001||float||The lower bound on the learning rate.|
|2||upper||0.006||float||The upper bound on the learning rate.|
|3||steps||100||int||The number of steps in every half cycle.|
|4||decay||0.99994||float||The exponential decay factor to decrease the learning rate by every step.|
use Rubix\ML\NeuralNet\Optimizers\Cyclical; $optimizer = new Cyclical(0.001, 0.005, 1000);
- L. N. Smith. (2017). Cyclical Learning Rates for Training Neural Networks.