Timezone: »

 
Poster
Diverse Weight Averaging for Out-of-Distribution Generalization
Alexandre Rame · Matthieu Kirchmeyer · Thibaud Rahier · Alain Rakotomamonjy · Patrick Gallinari · Matthieu Cord

Thu Dec 01 02:00 PM -- 04:00 PM (PST) @ Hall J #441
in Poster Session 6 »

Standard neural networks struggle to generalize under distribution shifts in computer vision. Fortunately, combining multiple networks can consistently improve out-of-distribution generalization. In particular, weight averaging (WA) strategies were shown to perform best on the competitive DomainBed benchmark; they directly average the weights of multiple networks despite their nonlinearities. In this paper, we propose Diverse Weight Averaging (DiWA), a new WA strategy whose main motivation is to increase the functional diversity across averaged models. To this end, DiWA averages weights obtained from several independent training runs: indeed, models obtained from different runs are more diverse than those collected along a single run thanks to differences in hyperparameters and training procedures. We motivate the need for diversity by a new bias-variance-covariance-locality decomposition of the expected error, exploiting similarities between WA and standard functional ensembling. Moreover, this decomposition highlights that WA succeeds when the variance term dominates, which we show occurs when the marginal distribution changes at test time. Experimentally, DiWA consistently improves the state of the art on DomainBed without inference overhead.

Keywords: Deep Learning · Computer Vision · distribution shifts · out-of-distribution generalization · Ensembling · Domain generalization · Weight Averaging
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Alexandre Rame (FAIR Meta AI - ISIR)

Currently research intern at FAIR Meta AI. PhD student at Sorbonne University in Paris under the supervision of Professor Matthieu Cord. Trying to make deep neural networks generalize out of distribution.

Matthieu Kirchmeyer (Sorbonne Université & Criteo)
Thibaud Rahier (Criteo AI Lab)

Ecole polytechnique graduate (Diplome d'ingenieur polytechnicien). Major: Applied Mathematics, Minors: Mathematics and Computer Science UC Berkeley graduate (M.A. in Statistics) PhD in Machine Learning (cifre) between INRIA and Schneider Electric in Grenoble, France Researcher at Criteo AI Lab in Grenoble, France

Alain Rakotomamonjy (Université de Rouen Normandie Criteo AI Lab)
Patrick Gallinari (Sorbonne Universite, Criteo AI Lab)
Matthieu Cord (Sorbonne University)

More from the Same Authors