Timezone: »
Out-of-distribution robustness: Limited image exposure of a four-year-old is enough to outperform ResNet-50
Lukas Huber · Robert Geirhos · Felix A. Wichmann
Mon Dec 13 09:00 AM -- 10:00 AM (PST) @
Event URL: https://openreview.net/forum?id=7yMg2rS9N5I »
Recent gains in model robustness towards out-of-distribution images are predominantly achieved through ever-increasing large-scale datasets. While this approach is very effective in achieving human-level distortion robustness, it raises the question of whether human robustness, too, requires massive amounts of experience. We therefore investigated the developmental trajectory of human object recognition robustness by comparing children aged 4–6, 7–9, 10–12, 13–15 against adults and against different neural networks. Assessing how recognition accuracy degrades when images are distorted by salt-and-pepper noise, we find that while overall performance improves with age, even the youngest children in the study showed remarkable robustness and outperformed standard CNNs on moderately distorted images. Using a back-of-the-envelope calculation, we then estimated the number of `images' that those young children had been exposed to during their lifetime. Even if we assume that a new image is seen every 2 seconds of wake time, children aged 4--6 only saw approximately 50M images, which is already lower than 90 epochs $\times$ 1.3M images during standard ImageNet training. This indicates that human out-of-distribution robustness develops very early and may not require seeing billions of different images during lifetime given the right choice of representation and information processing optimised during evolution.
Recent gains in model robustness towards out-of-distribution images are predominantly achieved through ever-increasing large-scale datasets. While this approach is very effective in achieving human-level distortion robustness, it raises the question of whether human robustness, too, requires massive amounts of experience. We therefore investigated the developmental trajectory of human object recognition robustness by comparing children aged 4–6, 7–9, 10–12, 13–15 against adults and against different neural networks. Assessing how recognition accuracy degrades when images are distorted by salt-and-pepper noise, we find that while overall performance improves with age, even the youngest children in the study showed remarkable robustness and outperformed standard CNNs on moderately distorted images. Using a back-of-the-envelope calculation, we then estimated the number of `images' that those young children had been exposed to during their lifetime. Even if we assume that a new image is seen every 2 seconds of wake time, children aged 4--6 only saw approximately 50M images, which is already lower than 90 epochs $\times$ 1.3M images during standard ImageNet training. This indicates that human out-of-distribution robustness develops very early and may not require seeing billions of different images during lifetime given the right choice of representation and information processing optimised during evolution.
Author Information
Lukas Huber (University of Tübingen)
Robert Geirhos (University of Tübingen)
Felix A. Wichmann (University of Tübingen)
More from the Same Authors
-
2021 Spotlight: How Well do Feature Visualizations Support Causal Understanding of CNN Activations? »
Roland S. Zimmermann · Judy Borowski · Robert Geirhos · Matthias Bethge · Thomas Wallis · Wieland Brendel -
2021 : ImageNet suffers from dichotomous data difficulty »
Kristof Meding · Luca Schulze Buschoff · Robert Geirhos · Felix A. Wichmann -
2021 Poster: How Well do Feature Visualizations Support Causal Understanding of CNN Activations? »
Roland S. Zimmermann · Judy Borowski · Robert Geirhos · Matthias Bethge · Thomas Wallis · Wieland Brendel -
2021 Oral: Partial success in closing the gap between human and machine vision »
Robert Geirhos · Kantharaju Narayanappa · Benjamin Mitzkus · Tizian Thieringer · Matthias Bethge · Felix A. Wichmann · Wieland Brendel -
2021 Poster: Partial success in closing the gap between human and machine vision »
Robert Geirhos · Kantharaju Narayanappa · Benjamin Mitzkus · Tizian Thieringer · Matthias Bethge · Felix A. Wichmann · Wieland Brendel -
2020 Poster: Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistency »
Robert Geirhos · Kristof Meding · Felix A. Wichmann -
2019 Poster: Perceiving the arrow of time in autoregressive motion »
Kristof Meding · Dominik Janzing · Bernhard Schölkopf · Felix A. Wichmann -
2019 Spotlight: Perceiving the arrow of time in autoregressive motion »
Kristof Meding · Dominik Janzing · Bernhard Schölkopf · Felix A. Wichmann -
2018 Poster: Generalisation in humans and deep neural networks »
Robert Geirhos · Carlos R. M. Temme · Jonas Rauber · Heiko H. Schütt · Matthias Bethge · Felix A. Wichmann