Timezone: »

 
Poster
Active Surrogate Estimators: An Active Learning Approach to Label-Efficient Model Evaluation
Jannik Kossen · Sebastian Farquhar · Yarin Gal · Thomas Rainforth

Tue Nov 29 09:00 AM -- 11:00 AM (PST) @ Hall J #114
in Poster Session 1 »

We propose Active Surrogate Estimators (ASEs), a new method for label-efficient model evaluation. Evaluating model performance is a challenging and important problem when labels are expensive. ASEs address this active testing problem using a surrogate-based estimation approach that interpolates the errors of points with unknown labels, rather than forming a Monte Carlo estimator. ASEs actively learn the underlying surrogate, and we propose a novel acquisition strategy, XWED, that tailors this learning to the final estimation task. We find that ASEs offer greater label-efficiency than the current state-of-the-art when applied to challenging model evaluation problems for deep neural networks.

Keywords: Active Learning · experimental design · sample-efficiency · bayesian active learning · active evaluation · active testing · model evaluation
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Jannik Kossen (University of Oxford)
Sebastian Farquhar (DeepMind)
Yarin Gal (University of OXford)
Thomas Rainforth (University of Oxford)

More from the Same Authors