Reliability Maps: A Tool to Enhance Probability Estimates and Improve Classification Accuracy
Abstract
We propose a general method to assess the reliability of two-class probabilities in an instance-wise manner. This is relevant, for instance, for obtaining calibrated multi-class probabilities from two-class probability scores. The LS-ECOC method approaches this by performing least-squares fitting over a suitable error-correcting output code matrix, where the optimisation resolves potential conflicts in the input probabilities. While this gives all input probabilities equal weight, we would like to spend less effort fitting unreliable probability estimates. We introduce the concept of a reliability map to accompany the more conventional notion of calibration map; and LS-ECOC-R which modifies LS-ECOC to take reliability into account. We demonstrate on synthetic data that this gets us closer to the Bayes-optimal classifier, even if the base classifiers are linear and hence have high bias. Results on UCI data sets demonstrate that multi-class accuracy also improves.
Cite
Text
Kull and Flach. "Reliability Maps: A Tool to Enhance Probability Estimates and Improve Classification Accuracy." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2014. doi:10.1007/978-3-662-44851-9_2Markdown
[Kull and Flach. "Reliability Maps: A Tool to Enhance Probability Estimates and Improve Classification Accuracy." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2014.](https://mlanthology.org/ecmlpkdd/2014/kull2014ecmlpkdd-reliability/) doi:10.1007/978-3-662-44851-9_2BibTeX
@inproceedings{kull2014ecmlpkdd-reliability,
title = {{Reliability Maps: A Tool to Enhance Probability Estimates and Improve Classification Accuracy}},
author = {Kull, Meelis and Flach, Peter A.},
booktitle = {European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases},
year = {2014},
pages = {18-33},
doi = {10.1007/978-3-662-44851-9_2},
url = {https://mlanthology.org/ecmlpkdd/2014/kull2014ecmlpkdd-reliability/}
}