Incremental Uncertainty-Aware Performance Monitoring with Labeling Intervention
Abstract
We study the problem of monitoring machine learning models under temporal distribution shifts, where circumstances change gradually over time, often leading to unnoticed yet significant declines in accuracy. We propose Incremental Uncertainty-aware Performance Monitoring (IUPM), a novel label-free method that estimates model performance by modeling time-dependent shifts using optimal transport. IUPM also quantifies uncertainty in performance estimates and introduces an active labeling strategy to reduce this uncertainty. We further showcase the benefits of IUPM on different datasets and simulated temporal shifts over existing baselines.
Cite
Text
Koebler et al. "Incremental Uncertainty-Aware Performance Monitoring with Labeling Intervention." NeurIPS 2024 Workshops: BDU, 2024.Markdown
[Koebler et al. "Incremental Uncertainty-Aware Performance Monitoring with Labeling Intervention." NeurIPS 2024 Workshops: BDU, 2024.](https://mlanthology.org/neuripsw/2024/koebler2024neuripsw-incremental/)BibTeX
@inproceedings{koebler2024neuripsw-incremental,
title = {{Incremental Uncertainty-Aware Performance Monitoring with Labeling Intervention}},
author = {Koebler, Alexander and Decker, Thomas and Thon, Ingo and Tresp, Volker and Buettner, Florian},
booktitle = {NeurIPS 2024 Workshops: BDU},
year = {2024},
url = {https://mlanthology.org/neuripsw/2024/koebler2024neuripsw-incremental/}
}