Rate-Oriented Point-Wise Confidence Bounds for ROC Curves
Abstract
Common approaches to generating confidence bounds around ROC curves have several shortcomings. We resolve these weaknesses with a new ‘rate-oriented’ approach. We generate confidence bounds composed of a series of confidence intervals for a consensus curve, each at a particular predicted positive rate (PPR), with the aim that each confidence interval contains new samples of this consensus curve with probability 95%. We propose two approaches; a parametric and a bootstrapping approach, which we base on a derivation from first principles. Our method is particularly appropriate with models used for a common type of task that we call rate-constrained, where a certain proportion of examples needs to be classified as positive by the model, such that the operating point will be set at a particular PPR value.
Cite
Text
Millard et al. "Rate-Oriented Point-Wise Confidence Bounds for ROC Curves." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2014. doi:10.1007/978-3-662-44851-9_26Markdown
[Millard et al. "Rate-Oriented Point-Wise Confidence Bounds for ROC Curves." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2014.](https://mlanthology.org/ecmlpkdd/2014/millard2014ecmlpkdd-rateoriented/) doi:10.1007/978-3-662-44851-9_26BibTeX
@inproceedings{millard2014ecmlpkdd-rateoriented,
title = {{Rate-Oriented Point-Wise Confidence Bounds for ROC Curves}},
author = {Millard, Louise A. C. and Kull, Meelis and Flach, Peter A.},
booktitle = {European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases},
year = {2014},
pages = {404-421},
doi = {10.1007/978-3-662-44851-9_26},
url = {https://mlanthology.org/ecmlpkdd/2014/millard2014ecmlpkdd-rateoriented/}
}