Proactive Detection of Model Degradation in Financial Fraud Prediction with Delayed Labels

Abstract

Financial fraud detection systems rely on machine learning models, but their performance degrades over time due to concept and covariate drift. A critical challenge is the delayed label problem: ground truth labels (confirming fraud) often arrive 1–6 months after the initial prediction. This creates a “blind period” where models can silently deteriorate, leading to substantial financial losses. Existing monitoring approaches, relying on delayed labels or statistical drift detection, are often too slow or insensitive. To address this, we propose PRODEM (PROactive DEtection of Model degradation), a framework that detects model degradation without immediate ground truth. PRODEM uses a meta-modeling technique: a sophisticated “meta-model” learns to predict when the deployed “primary” fraud model will make errors. We use a reverse distillation approach, where the meta-model specifically targets error prediction in out-of-time scenarios typical of fraud detection. Experiments on two proprietary datasets from a payment network show that PRODEM significantly improves degradation detection compared to statistical methods and recent drift detection techniques. Importantly, PRODEM identifies failing models before ground truth labels become available, mitigating the financial impact of model degradation in high-stakes decision-making. We also demonstrate PRODEM’s effectiveness at identifying increases in false positive rates, a crucial but often overlooked aspect of fraud model monitoring.