Timezone: »

 
Poster
Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks
Yuanzhi Li · Colin Wei · Tengyu Ma

Tue Dec 10 10:45 AM -- 12:45 PM (PST) @ East Exhibition Hall B + C #144
in Deep Learning -- Optimization for Deep Networks »

Stochastic gradient descent with a large initial learning rate is widely used for training modern neural net architectures. Although a small initial learning rate allows for faster training and better test performance initially, the large learning rate achieves better generalization soon after the learning rate is annealed. Towards explaining this phenomenon, we devise a setting in which we can prove that a two layer network trained with large initial learning rate and annealing provably generalizes better than the same network trained with a small learning rate from the start. The key insight in our analysis is that the order of learning different types of patterns is crucial: because the small learning rate model first memorizes easy-to-generalize, hard-to-fit patterns, it generalizes worse on hard-to-generalize, easier-to-fit patterns than its large learning rate counterpart. This concept translates to a larger-scale setting: we demonstrate that one can add a small patch to CIFAR-10 images that is immediately memorizable by a model with small initial learning rate, but ignored by the model with large learning rate until after annealing. Our experiments show that this causes the small learning rate model's accuracy on unmodified images to suffer, as it relies too much on the patch early on.

Keywords: Theory · Regularization · Deep Learning; Deep Learning -> Optimization for Deep Networks; Theory
[ Paper]  [ Slides] 

Author Information

Yuanzhi Li (Princeton)
Colin Wei (Stanford University)
Tengyu Ma (Stanford University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors