Timezone: »
Automatically discovering failures in vision models under real-world settings remains an open challenge. This work describes how off-the-shelf, large-scale, image-to-text and text-to-image models, trained on vast amounts of data, can be leveraged to automatically find such failures. We detail a pipeline that demonstrates how we can interrogate classifiers trained on ImageNet to find specific failure cases and discover spurious correlations. We also show that we can scale our approach to generate adversarial datasets targeting specific classifier architectures. This work serves as a proof-of-concept demonstrating the utility of large-scale generative models to automatically discover bugs in vision models in an open-ended manner.
Author Information
Olivia Wiles (DeepMind)
Isabela Albuquerque (DeepMind)
Sven Gowal (DeepMind)
More from the Same Authors
-
2021 Spotlight: Make Sure You're Unsure: A Framework for Verifying Probabilistic Specifications »
Leonard Berrada · Sumanth Dathathri · Krishnamurthy Dvijotham · Robert Stanforth · Rudy Bunel · Jonathan Uesato · Sven Gowal · M. Pawan Kumar -
2021 : Randomly projecting out distribution shifts for improved robustness »
Isabela Albuquerque · Joao Monteiro · Tiago H Falk -
2021 : A fine-grained analysis of robustness to distribution shifts »
Olivia Wiles · Sven Gowal · Florian Stimberg · Sylvestre-Alvise Rebuffi · Ira Ktena · Krishnamurthy Dvijotham · Taylan Cemgil -
2021 Poster: Make Sure You're Unsure: A Framework for Verifying Probabilistic Specifications »
Leonard Berrada · Sumanth Dathathri · Krishnamurthy Dvijotham · Robert Stanforth · Rudy Bunel · Jonathan Uesato · Sven Gowal · M. Pawan Kumar -
2021 Poster: Data Augmentation Can Improve Robustness »
Sylvestre-Alvise Rebuffi · Sven Gowal · Dan Andrei Calian · Florian Stimberg · Olivia Wiles · Timothy A Mann -
2021 Poster: Improving Robustness using Generated Data »
Sven Gowal · Sylvestre-Alvise Rebuffi · Olivia Wiles · Florian Stimberg · Dan Andrei Calian · Timothy A Mann -
2020 Poster: The Autoencoding Variational Autoencoder »
Taylan Cemgil · Sumedh Ghaisas · Krishnamurthy Dvijotham · Sven Gowal · Pushmeet Kohli -
2020 Spotlight: The Autoencoding Variational Autoencoder »
Taylan Cemgil · Sumedh Ghaisas · Krishnamurthy Dvijotham · Sven Gowal · Pushmeet Kohli -
2019 Poster: Adversarial Robustness through Local Linearization »
Chongli Qin · James Martens · Sven Gowal · Dilip Krishnan · Krishnamurthy Dvijotham · Alhussein Fawzi · Soham De · Robert Stanforth · Pushmeet Kohli -
2018 : Accepted papers »
Sven Gowal · Bogdan Kulynych · Marius Mosbach · Nicholas Frosst · Phil Roth · Utku Ozbulak · Simral Chaudhary · Toshiki Shibahara · Salome Viljoen · Nikita Samarin · Briland Hitaj · Rohan Taori · Emanuel Moss · Melody Guan · Lukas Schott · Angus Galloway · Anna Golubeva · Xiaomeng Jin · Felix Kreuk · Akshayvarun Subramanya · Vipin Pillai · Hamed Pirsiavash · Giuseppe Ateniese · Ankita Kalra · Logan Engstrom · Anish Athalye