A Unified Evaluation Framework for Epistemic Predictions
Abstract
Predictions of uncertainty-aware models are diverse, ranging from single point estimates (often averaged over prediction samples) to predictive distributions, to set-valued or credal-set representations. We propose a novel unified evaluation framework for uncertainty-aware classifiers, applicable to a wide range of model classes, which allows users to tailor the trade-off between accuracy and precision of predictions via a suitably designed performance metric. This makes possible the selection of the most suitable model for a particular real-world application as a function of the desired trade-off. Our experiments, concerning Bayesian, ensemble, evidential, deterministic, credal and belief function classifiers on the CIFAR-10, MNIST and CIFAR-100 datasets, show that the metric behaves as desired.
Cite
Text
Manchingal et al. "A Unified Evaluation Framework for Epistemic Predictions." Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, 2025.Markdown
[Manchingal et al. "A Unified Evaluation Framework for Epistemic Predictions." Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, 2025.](https://mlanthology.org/aistats/2025/manchingal2025aistats-unified/)BibTeX
@inproceedings{manchingal2025aistats-unified,
title = {{A Unified Evaluation Framework for Epistemic Predictions}},
author = {Manchingal, Shireen Kudukkil and Mubashar, Muhammad and Wang, Kaizheng and Cuzzolin, Fabio},
booktitle = {Proceedings of The 28th International Conference on Artificial Intelligence and Statistics},
year = {2025},
pages = {2017-2025},
volume = {258},
url = {https://mlanthology.org/aistats/2025/manchingal2025aistats-unified/}
}