Timezone: »

 
Poster
Calibrating Deep Neural Networks using Focal Loss
Jishnu Mukhoti · Viveka Kulharia · Amartya Sanyal · Stuart Golodetz · Philip Torr · Puneet Dokania

Thu Dec 10 09:00 AM -- 11:00 AM (PST) @ Poster Session 5 #1536
in Poster Session 6 »

Miscalibration -- a mismatch between a model's confidence and its correctness -- of Deep Neural Networks (DNNs) makes their predictions hard to rely on. Ideally, we want networks to be accurate, calibrated and confident. We show that, as opposed to the standard cross-entropy loss, focal loss (Lin et al., 2017) allows us to learn models that are already very well calibrated. When combined with temperature scaling, whilst preserving accuracy, it yields state-of-the-art calibrated models. We provide a thorough analysis of the factors causing miscalibration, and use the insights we glean from this to justify the empirically excellent performance of focal loss. To facilitate the use of focal loss in practice, we also provide a principled approach to automatically select the hyperparameter involved in the loss function. We perform extensive experiments on a variety of computer vision and NLP datasets, and with a wide variety of network architectures, and show that our approach achieves state-of-the-art calibration without compromising on accuracy in almost all cases. Code is available at https://github.com/torrvision/focal_calibration.

Author Information

Jishnu Mukhoti (University of Oxford)
Viveka Kulharia (University of Oxford)

I am a DPhil (PhD) student in the Torr Vision Group at the University of Oxford working under the supervision of Prof. Philip H. S. Torr and Dr. Puneet K. Dokania.

Amartya Sanyal (University of Oxford)
Stuart Golodetz (University of Oxford)

Stuart Golodetz is a Postdoctoral Research Associate working for Professors Niki Trigoni and Andrew Markham in the University of Oxford’s Department of Computer Science. He was previously the Director of FiveAI’s Oxford Research Group from 2018-20. He obtained his DPhil in Computer Science in 2011, working on 3D image segmentation and feature identification. He then spent two years in industry, working for SunGard in the area of credit risk management and for Semmle in the areas of logic programming and software analytics. After returning to academia in 2013, he spent a year working for the Smart Specs project of Dr. Stephen Hicks in the Nuffield Department of Clinical Neurosciences, and then four years in the Department of Engineering Science’s Torr Vision Group, before moving to FiveAI in 2018. His areas of interest include computer vision, SLAM, medical image analysis, computer games development and the intricacies of different programming languages, especially C++. He taught Computer Science as a Stipendiary Lecturer at Hertford College from 2014 to 2017. He is a member of the Association of C and C++ Users (ACCU), for whose magazines he has written a variety of articles, and also of Oxford Model Flying Club (OMFC).

Philip Torr (University of Oxford)
Puneet Dokania (Five AI and University of Oxford)

More from the Same Authors