Timezone: »

 
Poster
Soft Calibration Objectives for Neural Networks
Archit Karandikar · Nicholas Cain · Dustin Tran · Balaji Lakshminarayanan · Jonathon Shlens · Michael Mozer · Becca Roelofs

Wed Dec 08 04:30 PM -- 06:00 PM (PST) @
in Poster Session 4 »

Optimal decision making requires that classifiers produce uncertainty estimates consistent with their empirical accuracy. However, deep neural networks are often under- or over-confident in their predictions. Consequently, methods have been developed to improve the calibration of their predictive uncertainty both during training and post-hoc. In this work, we propose differentiable losses to improve calibration based on a soft (continuous) version of the binning operation underlying popular calibration-error estimators. When incorporated into training, these soft calibration losses achieve state-of-the-art single-model ECE across multiple datasets with less than 1% decrease in accuracy. For instance, we observe an 82% reduction in ECE (70% relative to the post-hoc rescaled ECE) in exchange for a 0.7% relative decrease in accuracy relative to the cross entropy baseline on CIFAR-100.When incorporated post-training, the soft-binning-based calibration error objective improves upon temperature scaling, a popular recalibration method. Overall, experiments across losses and datasets demonstrate that using calibration-sensitive procedures yield better uncertainty estimates under dataset shift than the standard practice of using a cross entropy loss and post-hoc recalibration methods.

Keywords: Deep Learning · Machine Learning

Author Information

Archit Karandikar (Google)
Nicholas Cain (Allen Institute for Brain Science)
Dustin Tran (Google Brain)
Balaji Lakshminarayanan (Google Brain)
Jonathon Shlens (Google)
Michael Mozer (Google Research / University of Colorado)
Becca Roelofs (Google Research)

More from the Same Authors