When Is Multicalibration Post-Processing Necessary?

Abstract

Calibration is a well-studied property of predictors which guarantees meaningful uncertainty estimates. Multicalibration is a related notion --- originating in algorithmic fairness --- which requires predictors to be simultaneously calibrated over a potentially complex and overlapping collection of protected subpopulations (such as groups defined by ethnicity, race, or income). We conduct the first comprehensive study evaluating the usefulness of multicalibration post-processing across a broad set of tabular, image, and language datasets for models spanning from simple decision trees to 90 million parameter fine-tuned LLMs. Our findings can be summarized as follows: (1) models which are calibrated out of the box tend to be relatively multicalibrated without any additional post-processing; (2) multicalibration can help inherently uncalibrated models and also large vision and language models; and (3) traditional calibration measures may sometimes provide multicalibration implicitly. More generally, we also distill many independent observations which may be useful for practical and effective applications of multicalibration post-processing in real-world contexts.

Cite

Text

Hansen et al. "When Is Multicalibration Post-Processing Necessary?." Neural Information Processing Systems, 2024. doi:10.52202/079017-1213

Markdown

[Hansen et al. "When Is Multicalibration Post-Processing Necessary?." Neural Information Processing Systems, 2024.](https://mlanthology.org/neurips/2024/hansen2024neurips-multicalibration/) doi:10.52202/079017-1213

BibTeX

@inproceedings{hansen2024neurips-multicalibration,
  title     = {{When Is Multicalibration Post-Processing Necessary?}},
  author    = {Hansen, Dutch and Devic, Siddartha and Nakkiran, Preetum and Sharan, Vatsal},
  booktitle = {Neural Information Processing Systems},
  year      = {2024},
  doi       = {10.52202/079017-1213},
  url       = {https://mlanthology.org/neurips/2024/hansen2024neurips-multicalibration/}
}