Timezone: »
Stochastic Pruning: Fine-Tuning, and PAC-Bayes bound optimization
Soufiane Hayou · Bobby He · Gintare Karolina Dziugaite
Event URL: https://openreview.net/forum?id=at_dDR2u6K »
We introduce an algorithmic framework for stochastic fine-tuning of pruning masks, starting from masks produced by several baselines. We further show that by minimizing a PAC-Bayes bound with data-dependent priors, we obtain a self-bounded learning algorithm with numerically tight bounds. In the linear model, we show that a PAC-Bayes generalization error bound is controlled by the magnitude of the change in feature alignment between the prior'' and
posterior'' data.
Author Information
Soufiane Hayou (National University of Singapore)
PTA Assistant Professor of Mathematics at NUS
Bobby He (University of Oxford)
Gintare Karolina Dziugaite (Google Research, Brain Team)
More from the Same Authors
-
2021 Spotlight: Towards a Unified Information-Theoretic Framework for Generalization »
Mahdi Haghifam · Gintare Karolina Dziugaite · Shay Moran · Dan Roy -
2021 : The Dynamics of Functional Diversity throughout Neural Network Training »
Lee Zamparo · Marc-Etienne Brunet · Thomas George · Sepideh Kharaghani · Gintare Karolina Dziugaite -
2021 : Uncertainty Quantification in End-to-End Implicit Neural Representations for Medical Imaging »
Francisca Vasconcelos · Bobby He · Yee Teh -
2022 : The curse of (non)convexity: The case of an Optimization-Inspired Data Pruning algorithm »
Fadhel Ayed · Soufiane Hayou -
2021 : Spotlight Talk 5 »
Soufiane Hayou -
2021 Poster: Regularization in ResNet with Stochastic Depth »
Soufiane Hayou · Fadhel Ayed -
2021 Poster: Deep Learning on a Data Diet: Finding Important Examples Early in Training »
Mansheej Paul · Surya Ganguli · Gintare Karolina Dziugaite -
2021 Poster: Towards a Unified Information-Theoretic Framework for Generalization »
Mahdi Haghifam · Gintare Karolina Dziugaite · Shay Moran · Dan Roy -
2018 Poster: Data-dependent PAC-Bayes priors via differential privacy »
Gintare Karolina Dziugaite · Daniel Roy