Timezone: »

Cross-validation Confidence Intervals for Test Error
Pierre Bayle · Alexandre Bayle · Lucas Janson · Lester Mackey

Tue Dec 08 09:00 AM -- 11:00 AM (PST) @ Poster Session 1 #190

This work develops central limit theorems for cross-validation and consistent estimators of the asymptotic variance under weak stability conditions on the learning algorithm. Together, these results provide practical, asymptotically-exact confidence intervals for k-fold test error and valid, powerful hypothesis tests of whether one learning algorithm has smaller k-fold test error than another. These results are also the first of their kind for the popular choice of leave-one-out cross-validation. In our experiments with diverse learning algorithms, the resulting intervals and tests outperform the most popular alternative methods from the literature.

Author Information

Pierre Bayle (Princeton University)
Alexandre Bayle (Harvard University)
Lucas Janson (Harvard University)
Lester Mackey (Microsoft Research)

More from the Same Authors