Beyond Top-Class Agreement: Using Divergences to Forecast Performance Under Distribution Shift

Abstract

Knowing if a model will generalize to data `in the wild' is crucial for safe deployment. To this end, we study model disagreement notions that consider the full predictive distribution - specifically disagreement based on Hellinger distance, Jensen-Shannon and Kullback–Leibler divergence. We find that divergence-based scores provide better test error estimates and detection rates on out-of-distribution data compared to their top-1 counterparts. Experiments involve standard vision and foundation models.

PDF NeurIPSW OpenReview Semantic Scholar

Cite

Text

Schirmer et al. "Beyond Top-Class Agreement: Using Divergences to Forecast Performance Under Distribution Shift." NeurIPS 2023 Workshops: DistShift, 2023.

Markdown

[Schirmer et al. "Beyond Top-Class Agreement: Using Divergences to Forecast Performance Under Distribution Shift." NeurIPS 2023 Workshops: DistShift, 2023.](https://mlanthology.org/neuripsw/2023/schirmer2023neuripsw-beyond/)

BibTeX

@inproceedings{schirmer2023neuripsw-beyond,
  title     = {{Beyond Top-Class Agreement: Using Divergences to Forecast Performance Under Distribution Shift}},
  author    = {Schirmer, Mona and Zhang, Dan and Nalisnick, Eric},
  booktitle = {NeurIPS 2023 Workshops: DistShift},
  year      = {2023},
  url       = {https://mlanthology.org/neuripsw/2023/schirmer2023neuripsw-beyond/}
}