Timezone: »

 
A Better Way to Decay: Proximal Gradient Training Algorithms for Neural Nets
Liu Yang · Jifan Zhang · Joseph Shenouda · Dimitris Papailiopoulos · Kangwook Lee · Robert Nowak

@
in OPT 2022: Optimization for Machine Learning »
Event URL: https://openreview.net/forum?id=4y1xh8jClhC »
Weight decay is one of the most widely used forms of regularization in deep learning, and has been shown to improve generalization and robustness. The optimization objective driving weight decay is a sum of losses plus a term proportional to the sum of squared weights. This paper argues that stochastic gradient descent (SGD) may be an inefficient algorithm for this objective. For neural networks with ReLU activations, solutions to the weight decay objective are equivalent to those of a different objective in which the regularization term is instead a sum of products of $\ell_2$ (not squared) norms of the input and output weights associated each ReLU. This alternative \emph{(and effectively equivalent)} regularization suggests a novel proximal gradient algorithm for network training. Theory and experiments support the new training approach, showing that it can converge much faster to the \emph{sparse} solutions it shares with standard weight decay training.

Author Information

Liu Yang (University of Wisconsin, Madison)
Jifan Zhang (University of Wisconsin)
Joseph Shenouda (University of Wisconsin Madison)
Dimitris Papailiopoulos (University of Wisconsin-Madison)
Kangwook Lee (UW Madison, Krafton)
Robert Nowak (University of Wisconsion-Madison)

More from the Same Authors