Timezone: »

 
The impact of domain shift on the calibration of fine-tuned models
Jay Mohta · Colin Raffel

@
in Distribution shifts: connecting methods and applications (DistShift) »
Event URL: https://openreview.net/forum?id=dZ7MVojplmi »

Transfer learning has become a standard technique in computer vision and natural language processing thanks to the fact that it often substantially improves performance on downstream tasks. Recent work by Hendrycks et al. demonstrated that using a pre-trained model can also significantly improve a model's calibration, i.e. how well the model's confidence estimates correspond to the probability of its prediction being correct. In this paper, we provide some nuance to the claim that pre-training improves calibration by demonstrating that this beneficial effect diminishes when there is a domain shift between the pre-training and fine-tuning tasks.

Author Information

Jay Mohta (North Carolina State University)
Colin Raffel (UNC Chapel Hill and Hugging Face)

More from the Same Authors