Calibrating Expressions of Certainty
Abstract
We present a novel approach to calibrating linguistic expressions of certainty, e.g., "Maybe" and "Likely". Unlike prior work that assigns a single score to each certainty phrase, we model uncertainty as distributions over the simplex to capture their semantics more accurately. To accommodate this new representation of certainty, we generalize existing measures of miscalibration and introduce a novel post-hoc calibration method. Leveraging these tools, we analyze the calibration of both humans (e.g., radiologists) and computational models (e.g., language models) and provide interpretable suggestions to improve their calibration.
Cite
Text
Wang et al. "Calibrating Expressions of Certainty." International Conference on Learning Representations, 2025.Markdown
[Wang et al. "Calibrating Expressions of Certainty." International Conference on Learning Representations, 2025.](https://mlanthology.org/iclr/2025/wang2025iclr-calibrating/)BibTeX
@inproceedings{wang2025iclr-calibrating,
title = {{Calibrating Expressions of Certainty}},
author = {Wang, Peiqi and Lam, Barbara D. and Liu, Yingcheng and Asgari-Targhi, Ameneh and Panda, Rameswar and Wells, William M and Kapur, Tina and Golland, Polina},
booktitle = {International Conference on Learning Representations},
year = {2025},
url = {https://mlanthology.org/iclr/2025/wang2025iclr-calibrating/}
}