Timezone: »

 
Poster
Pruning’s Effect on Generalization Through the Lens of Training and Regularization
Tian Jin · Michael Carbin · Dan Roy · Jonathan Frankle · Gintare Karolina Dziugaite

Tue Nov 29 02:00 PM -- 04:00 PM (PST) @ Hall J #715
in Poster Session 2 »

Practitioners frequently observe that pruning improves model generalization. A long-standing hypothesis based on bias-variance trade-off attributes this generalization improvement to model size reduction. However, recent studies on over-parameterization characterize a new model size regime, in which larger models achieve better generalization. Pruning models in this over-parameterized regime leads to a contradiction -- while theory predicts that reducing model size harms generalization, pruning to a range of sparsities nonetheless improves it. Motivated by this contradiction, we re-examine pruning’s effect on generalization empirically.We show that size reduction cannot fully account for the generalization-improving effect of standard pruning algorithms. Instead, we find that pruning leads to better training at specific sparsities, improving the training loss over the dense model. We find that pruning also leads to additional regularization at other sparsities, reducing the accuracy degradation due to noisy examples over the dense model. Pruning extends model training time and reduces model size. These two factors improve training and add regularization respectively. We empirically demonstrate that both factors are essential to fully explaining pruning's impact on generalization.

Keywords: generalization · Pruning · empirical deep learning

Author Information

Tian Jin (Massachusetts Institute of Technology)
Michael Carbin (MIT)
Dan Roy (U of Toronto; Vector)
Jonathan Frankle (School of Engineering and Applied Sciences, Harvard University)
Gintare Karolina Dziugaite (Google Research, Brain Team)

More from the Same Authors