Timezone: »

 
Poster
Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensembles
Jiefeng Chen · Frederick Liu · Besim Avci · Xi Wu · Yingyu Liang · Somesh Jha

Wed Dec 08 04:30 PM -- 06:00 PM (PST) @
in Poster Session 4 »

When a deep learning model is deployed in the wild, it can encounter test data drawn from distributions different from the training data distribution and suffer drop in performance. For safe deployment, it is essential to estimate the accuracy of the pre-trained model on the test data. However, the labels for the test inputs are usually not immediately available in practice, and obtaining them can be expensive. This observation leads to two challenging tasks: (1) unsupervised accuracy estimation, which aims to estimate the accuracy of a pre-trained classifier on a set of unlabeled test inputs; (2) error detection, which aims to identify mis-classified test inputs. In this paper, we propose a principled and practically effective framework that simultaneously addresses the two tasks. The proposed framework iteratively learns an ensemble of models to identify mis-classified data points and performs self-training to improve the ensemble with the identified points. Theoretical analysis demonstrates that our framework enjoys provable guarantees for both accuracy estimation and error detection under mild conditions readily satisfied by practical deep learning models. Along with the framework, we proposed and experimented with two instantiations and achieved state-of-the-art results on 59 tasks. For example, on iWildCam, one instantiation reduces the estimation error for unsupervised accuracy estimation by at least 70% and improves the F1 score for error detection by at least 4.7% compared to existing methods.

Keywords: Deep Learning

Author Information

Jiefeng Chen (University of Wisconsin-Madison)

I am currently a third year Phd student at University of Wisconsin-Madison, in the Computer Science Department. I am co-advised by Prof. Yingyu Liang and Prof. Somesh Jha. I work on trustworthy machine learning with research questions like "How to make machine learning models produce stable explanations of their predictions?", "How to train models that produce robust predictions under adversarial perturbations?, and "Understand when and why some defense mechanisms work?". I obtained my Bachelor's degree in Computer Science from Shanghai Jiao Tong University (SJTU).

Frederick Liu (Google)
Besim Avci (Google)
Xi Wu (Google)
Yingyu Liang (Princeton University)
Somesh Jha (University of Wisconsin, Madison)

More from the Same Authors