Rate-Constrained Ranking and the Rate-Weighted AUC
Abstract
Ranking tasks, where instances are ranked by a predicted score, are common in machine learning. Often only a proportion of the instances in the ranking can be processed, and this quantity, the predicted positive rate (PPR), may not be known precisely. In this situation, the evaluation of a model’s performance needs to account for these imprecise constraints on the PPR, but existing metrics such as the area under the ROC curve (AUC) and early retrieval metrics such as normalised discounted cumulative gain (NDCG) cannot do this. In this paper we introduce a novel metric, the rate-weighted AUC (rAUC), to evaluate ranking models when constraints across the PPR exist, and provide an efficient algorithm to estimate the rAUC using an empirical ROC curve. Our experiments show that rAUC, AUC and NDCG often select different models. We demonstrate the usefulness of rAUC on a practical application: ranking articles for rapid reviews in epidemiology.
Cite
Text
Millard et al. "Rate-Constrained Ranking and the Rate-Weighted AUC." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2014. doi:10.1007/978-3-662-44851-9_25Markdown
[Millard et al. "Rate-Constrained Ranking and the Rate-Weighted AUC." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2014.](https://mlanthology.org/ecmlpkdd/2014/millard2014ecmlpkdd-rateconstrained/) doi:10.1007/978-3-662-44851-9_25BibTeX
@inproceedings{millard2014ecmlpkdd-rateconstrained,
title = {{Rate-Constrained Ranking and the Rate-Weighted AUC}},
author = {Millard, Louise A. C. and Flach, Peter A. and Higgins, Julian P. T.},
booktitle = {European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases},
year = {2014},
pages = {386-403},
doi = {10.1007/978-3-662-44851-9_25},
url = {https://mlanthology.org/ecmlpkdd/2014/millard2014ecmlpkdd-rateconstrained/}
}