Timezone: »

 
Poster
Assaying Out-Of-Distribution Generalization in Transfer Learning
Florian Wenzel · Andrea Dittadi · Peter Gehler · Carl-Johann Simon-Gabriel · Max Horn · Dominik Zietlow · David Kernert · Chris Russell · Thomas Brox · Bernt Schiele · Bernhard Schölkopf · Francesco Locatello

Thu Dec 01 09:00 AM -- 11:00 AM (PST) @ Hall J #623
in Poster Session 5 »

Since out-of-distribution generalization is a generally ill-posed problem, various proxy targets (e.g., calibration, adversarial robustness, algorithmic corruptions, invariance across shifts) were studied across different research programs resulting in different recommendations. While sharing the same aspirational goal, these approaches have never been tested under the same experimental conditions on real data. In this paper, we take a unified view of previous work, highlighting message discrepancies that we address empirically, and providing recommendations on how to measure the robustness of a model and how to improve it. To this end, we collect 172 publicly available dataset pairs for training and out-of-distribution evaluation of accuracy, calibration error, adversarial attacks, environment invariance, and synthetic corruptions. We fine-tune over 31k networks, from nine different architectures in the many- and few-shot setting. Our findings confirm that in- and out-of-distribution accuracies tend to increase jointly, but show that their relation is largely dataset-dependent, and in general more nuanced and more complex than posited by previous, smaller scale studies.

Keywords: distribution shifts · robustness · out-of-distribution generalization · large-scale empirical study
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Florian Wenzel (AWS)
Andrea Dittadi (Technical University of Denmark)
Peter Gehler (Amazon)
Carl-Johann Simon-Gabriel (Amazon Web Services)
Max Horn (ETH Zürich)
Dominik Zietlow (Max Planck Institute for Intelligent Systems, Max-Planck Institute)
David Kernert (Amazon)
Chris Russell (Amazon Web Services)
Thomas Brox (University of Freiburg)
Bernt Schiele (Max Planck Institute for Informatics)
Bernhard Schölkopf (MPI for Intelligent Systems, Tübingen)

Bernhard Scholkopf received degrees in mathematics (London) and physics (Tubingen), and a doctorate in computer science from the Technical University Berlin. He has researched at AT&T Bell Labs, at GMD FIRST, Berlin, at the Australian National University, Canberra, and at Microsoft Research Cambridge (UK). In 2001, he was appointed scientific member of the Max Planck Society and director at the MPI for Biological Cybernetics; in 2010 he founded the Max Planck Institute for Intelligent Systems. For further information, see www.kyb.tuebingen.mpg.de/~bs.

Francesco Locatello (Amazon)

More from the Same Authors