A rate that is too large can make optimization unstable, while a very small rate can make progress slow or stall in poor regions. Schedules often change the rate over the course of training.
The learning rate controls the size of parameter updates made by an optimizer during training.
A rate that is too large can make optimization unstable, while a very small rate can make progress slow or stall in poor regions. Schedules often change the rate over the course of training.