Timezone: »

 
Oral
Partial success in closing the gap between human and machine vision
Robert Geirhos · Kantharaju Narayanappa · Benjamin Mitzkus · Tizian Thieringer · Matthias Bethge · Felix A. Wichmann · Wieland Brendel

Wed Dec 08 08:00 AM -- 08:15 AM (PST) @
in Oral Session 3: Vision Applications »

A few years ago, the first CNN surpassed human performance on ImageNet. However, it soon became clear that machines lack robustness on more challenging test cases, a major obstacle towards deploying machines "in the wild" and towards obtaining better computational models of human visual perception. Here we ask: Are we making progress in closing the gap between human and machine vision? To answer this question, we tested human observers on a broad range of out-of-distribution (OOD) datasets, recording 85,120 psychophysical trials across 90 participants. We then investigated a range of promising machine learning developments that crucially deviate from standard supervised CNNs along three axes: objective function (self-supervised, adversarially trained, CLIP language-image training), architecture (e.g. vision transformers), and dataset size (ranging from 1M to 1B).Our findings are threefold. (1.) The longstanding distortion robustness gap between humans and CNNs is closing, with the best models now exceeding human feedforward performance on most of the investigated OOD datasets. (2.) There is still a substantial image-level consistency gap, meaning that humans make different errors than models. In contrast, most models systematically agree in their categorisation errors, even substantially different ones like contrastive self-supervised vs. standard supervised models. (3.) In many cases, human-to-model consistency improves when training dataset size is increased by one to three orders of magnitude. Our results give reason for cautious optimism: While there is still much room for improvement, the behavioural difference between human and machine vision is narrowing. In order to measure future progress, 17 OOD datasets with image-level human behavioural data and evaluation code are provided as a toolbox and benchmark at: https://github.com/bethgelab/model-vs-human/

Keywords: Machine Learning · Robustness · Transformers

Author Information

Robert Geirhos (University of Tübingen)
Kantharaju Narayanappa (University of Tuebingen)
Benjamin Mitzkus (University of Tuebingen)
Tizian Thieringer (University of Tuebingen)
Matthias Bethge (University of Tübingen)
Felix A. Wichmann (University of Tübingen)
Wieland Brendel (AG Bethge, University of Tübingen)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors