Timezone: »

 
Poster
Revisiting the Calibration of Modern Neural Networks
Matthias Minderer · Josip Djolonga · Rob Romijnders · Frances Hubis · Xiaohua Zhai · Neil Houlsby · Dustin Tran · Mario Lucic

Tue Dec 07 08:30 AM -- 10:00 AM (PST) @
in Poster Session 1 »

Accurate estimation of predictive uncertainty (model calibration) is essential for the safe application of neural networks. Many instances of miscalibration in modern neural networks have been reported, suggesting a trend that newer, more accurate models produce poorly calibrated predictions. Here, we revisit this question for recent state-of-the-art image classification models. We systematically relate model calibration and accuracy, and find that the most recent models, notably those not using convolutions, are among the best calibrated. Trends observed in prior model generations, such as decay of calibration with distribution shift or model size, are less pronounced in recent architectures. We also show that model size and amount of pretraining do not fully explain these differences, suggesting that architecture is a major determinant of calibration properties.

Keywords: Deep Learning · Machine Learning · Vision

Author Information

Matthias Minderer (Google Research)
Josip Djolonga (Google Research, Brain Team)
Rob Romijnders (University of Amsterdam)
Frances Hubis (Google Research Zurich)
Xiaohua Zhai (Google Brain)
Neil Houlsby (Google)
Dustin Tran (Google Brain)
Mario Lucic (Google Brain)

More from the Same Authors