Timezone: »
Robustness to distribution shifts is critical for deploying machine learning models in the real world. Despite this necessity, there has been little work in defining the underlying mechanisms that cause these shifts and evaluating the robustness of algorithms across multiple, different distribution shifts. To this end, we introduce a framework that enables fine-grained analysis of various distribution shifts. We provide a holistic analysis of current state-of-the-art methods by evaluating 19 distinct methods grouped into five categories across both synthetic and real-world datasets. Overall, we train more than 85K models. Our experimental framework can be easily extended to include new methods, shifts, and datasets. We find, unlike previous work [Gulrajani and Lopez-Paz, 2021], that progress has been made over a standard ERM baseline; in particular, pre-training and augmentations (learned or heuristic) offer large gains in many cases. However, the best methods are not consistent over different datasets and shifts.
Author Information
Olivia Wiles (DeepMind)
Sven Gowal (DeepMind)
Florian Stimberg (DeepMind)
Sylvestre-Alvise Rebuffi (University of Oxford)
Ira Ktena (DeepMind)
Krishnamurthy Dvijotham (DeepMind)
Krishnamurthy Dvijotham is a research scientist at Google Deepmind. Until recently, he was a research engineer at Pacific Northwest National Laboratory (PNNL) in the optimization and control group. He was previously a postdoctoral fellow at the Center for Mathematics of Information at Caltech. He received his PhD in computer science and engineering from the University of Washington, Seattle in 2014 and a bachelors from IIT Bombay in 2008. His research interests span stochastic control theory, artificial intelligence, machine learning and markets/economics, and his work is motivated primarily by problems arising in large-scale infrastructure systems like the power grid. His research has won awards at several conferences in optimization, AI and machine learning.
Taylan Cemgil (DeepMind)
More from the Same Authors
-
2021 Spotlight: Make Sure You're Unsure: A Framework for Verifying Probabilistic Specifications »
Leonard Berrada · Sumanth Dathathri · Krishnamurthy Dvijotham · Robert Stanforth · Rudy Bunel · Jonathan Uesato · Sven Gowal · M. Pawan Kumar -
2022 : Active Acquisition for Multimodal Temporal Data: A Challenging Decision-Making Task »
Jannik Kossen · Cătălina Cangea · Eszter Vértes · Andrew Jaegle · Viorica Patraucean · Ira Ktena · Nenad Tomasev · Danielle Belgrave -
2022 : Discovering Bugs in Vision Models using Off-the-shelf Image Generation and Captioning »
Olivia Wiles · Isabela Albuquerque · Sven Gowal -
2023 Poster: Affinity-Aware Graph Networks »
Ameya Velingker · Ali Sinop · Ira Ktena · Petar Veličković · Sreenivas Gollapudi -
2023 Poster: Provably Bounding Neural Network Preimages »
Christopher Brix · Suhas Kotha · Huan Zhang · J. Zico Kolter · Krishnamurthy Dvijotham -
2023 Poster: Training Private Models That Know What They Don’t Know »
Stephan Rabanser · Anvith Thudi · Abhradeep Guha Thakurta · Krishnamurthy Dvijotham · Nicolas Papernot -
2023 Poster: Learning to Receive Help: Intervention-Aware Concept Embedding Models »
Mateo Espinosa Zarlenga · Katie Collins · Krishnamurthy Dvijotham · Adrian Weller · Zohreh Shams · Mateja Jamnik -
2021 Poster: Make Sure You're Unsure: A Framework for Verifying Probabilistic Specifications »
Leonard Berrada · Sumanth Dathathri · Krishnamurthy Dvijotham · Robert Stanforth · Rudy Bunel · Jonathan Uesato · Sven Gowal · M. Pawan Kumar -
2021 Poster: Overcoming the Convex Barrier for Simplex Inputs »
Harkirat Singh Behl · M. Pawan Kumar · Philip Torr · Krishnamurthy Dvijotham -
2021 Poster: Data Augmentation Can Improve Robustness »
Sylvestre-Alvise Rebuffi · Sven Gowal · Dan Andrei Calian · Florian Stimberg · Olivia Wiles · Timothy A Mann -
2021 Poster: Improving Robustness using Generated Data »
Sven Gowal · Sylvestre-Alvise Rebuffi · Olivia Wiles · Florian Stimberg · Dan Andrei Calian · Timothy A Mann -
2020 Poster: The Autoencoding Variational Autoencoder »
Taylan Cemgil · Sumedh Ghaisas · Krishnamurthy Dvijotham · Sven Gowal · Pushmeet Kohli -
2020 Spotlight: The Autoencoding Variational Autoencoder »
Taylan Cemgil · Sumedh Ghaisas · Krishnamurthy Dvijotham · Sven Gowal · Pushmeet Kohli -
2019 Poster: Adversarial Robustness through Local Linearization »
Chongli Qin · James Martens · Sven Gowal · Dilip Krishnan · Krishnamurthy Dvijotham · Alhussein Fawzi · Soham De · Robert Stanforth · Pushmeet Kohli -
2018 : Accepted papers »
Sven Gowal · Bogdan Kulynych · Marius Mosbach · Nicholas Frosst · Phil Roth · Utku Ozbulak · Simral Chaudhary · Toshiki Shibahara · Salome Viljoen · Nikita Samarin · Briland Hitaj · Rohan Taori · Emanuel Moss · Melody Guan · Lukas Schott · Angus Galloway · Anna Golubeva · Xiaomeng Jin · Felix Kreuk · Akshayvarun Subramanya · Vipin Pillai · Hamed Pirsiavash · Giuseppe Ateniese · Ankita Kalra · Logan Engstrom · Anish Athalye -
2017 Poster: Learning multiple visual domains with residual adapters »
Sylvestre-Alvise Rebuffi · Hakan Bilen · Andrea Vedaldi -
2017 Spotlight: Learning multiple visual domains with residual adapters »
Sylvestre-Alvise Rebuffi · Hakan Bilen · Andrea Vedaldi -
2013 Poster: Summary Statistics for Partitionings and Feature Allocations »
Isik B Fidaner · Taylan Cemgil -
2011 Poster: Generalised Coupled Tensor Factorisation »
Kenan Y Yılmaz · Taylan Cemgil · Umut Simsekli