Timezone: »

 
Poster
Cross-validation Confidence Intervals for Test Error
Pierre Bayle · Alexandre Bayle · Lucas Janson · Lester Mackey

Tue Dec 08 09:00 AM -- 11:00 AM (PST) @ Poster Session 1 #190
in Poster Session 2 »

This work develops central limit theorems for cross-validation and consistent estimators of the asymptotic variance under weak stability conditions on the learning algorithm. Together, these results provide practical, asymptotically-exact confidence intervals for k-fold test error and valid, powerful hypothesis tests of whether one learning algorithm has smaller k-fold test error than another. These results are also the first of their kind for the popular choice of leave-one-out cross-validation. In our experiments with diverse learning algorithms, the resulting intervals and tests outperform the most popular alternative methods from the literature.

Keywords: Theory · Regularization · Deep Learning; Deep Learning -> Optimization for Deep Networks; Theory

Author Information

Pierre Bayle (Princeton University)
Alexandre Bayle (Harvard University)
Lucas Janson (Harvard University)
Lester Mackey (Microsoft Research)

More from the Same Authors