Cross-Validation Confidence Intervals for Test Error
Abstract
This work develops central limit theorems for cross-validation and consistent estimators of the asymptotic variance under weak stability conditions on the learning algorithm. Together, these results provide practical, asymptotically-exact confidence intervals for k-fold test error and valid, powerful hypothesis tests of whether one learning algorithm has smaller k-fold test error than another. These results are also the first of their kind for the popular choice of leave-one-out cross-validation. In our experiments with diverse learning algorithms, the resulting intervals and tests outperform the most popular alternative methods from the literature.
Cite
Text
Bayle et al. "Cross-Validation Confidence Intervals for Test Error." Neural Information Processing Systems, 2020.Markdown
[Bayle et al. "Cross-Validation Confidence Intervals for Test Error." Neural Information Processing Systems, 2020.](https://mlanthology.org/neurips/2020/bayle2020neurips-crossvalidation/)BibTeX
@inproceedings{bayle2020neurips-crossvalidation,
title = {{Cross-Validation Confidence Intervals for Test Error}},
author = {Bayle, Pierre and Bayle, Alexandre and Janson, Lucas and Mackey, Lester W.},
booktitle = {Neural Information Processing Systems},
year = {2020},
url = {https://mlanthology.org/neurips/2020/bayle2020neurips-crossvalidation/}
}