Revisiting the Evaluation of Uncertainty Estimation and Its Application to Explore Model Complexity-Uncertainty Trade-Off

Yukun Ding, Jinglan Liu, Jinjun Xiong, Yiyu Shi

CVPRW 2020 pp. 22-31

doi:10.1109/CVPRW50498.2020.00010 /cvprw/2020/ding2020cvprw-revisiting/

Abstract

Accurately estimating uncertainties in neural network predictions is of great importance in building trusted DNNs-based models, and there is an increasing interest in providing accurate uncertainty estimation on many tasks, such as security cameras and autonomous driving vehicles. In this paper, we focus on the two main use cases of uncertainty estimation, i.e., selective prediction and confidence calibration. We first reveal potential issues of commonly used quality metrics for uncertainty estimation in both use cases, and propose our new metrics to mitigate them. We then apply these new metrics to explore the trade-off between model complexity and uncertainty estimation quality, a critically missing work in the literature. Our empirical experiment results validate the superiority of the proposed metrics, and some interesting trends about the complexity-uncertainty trade-off are observed.

PDF CVPRW Semantic Scholar

Cite

Text

Ding et al. "Revisiting the Evaluation of Uncertainty Estimation and Its Application to Explore Model Complexity-Uncertainty Trade-Off." IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020. doi:10.1109/CVPRW50498.2020.00010

Markdown

[Ding et al. "Revisiting the Evaluation of Uncertainty Estimation and Its Application to Explore Model Complexity-Uncertainty Trade-Off." IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020.](https://mlanthology.org/cvprw/2020/ding2020cvprw-revisiting/) doi:10.1109/CVPRW50498.2020.00010

BibTeX

@inproceedings{ding2020cvprw-revisiting,
  title     = {{Revisiting the Evaluation of Uncertainty Estimation and Its Application to Explore Model Complexity-Uncertainty Trade-Off}},
  author    = {Ding, Yukun and Liu, Jinglan and Xiong, Jinjun and Shi, Yiyu},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops},
  year      = {2020},
  pages     = {22-31},
  doi       = {10.1109/CVPRW50498.2020.00010},
  url       = {https://mlanthology.org/cvprw/2020/ding2020cvprw-revisiting/}
}