Better Features, Better Calibration: A Simple Fix for Overconfident Networks

Abstract

A model is considered perfectly calibrated when the predicted probabilities align accurately with the true likelihood of the associated classes being correct. Past studies have shown that Deep Neural Networks (DNNs) are susceptible to overfitting and generate miscalibrated predictions. In this paper, we identify that the miscalibration problem in DNNs can be traced back to the features learned by the network. To this end, we propose a new training approach called RelCal , which guides the model to focus on a subset of relevant features for each class. Our empirical analysis highlights that training with RelCal helps mitigate overconfidence in DNNs, leading to better-calibrated models in terms of Expected Calibration Error (ECE) and Adaptive Expected Calibration Error (AECE). We demonstrate the state-of-the-art results on a diverse range of 8 image classification datasets across architectures spanning CNNs to Transformer-based architectures in terms of network calibration without compromising discriminative performance. Compared to the current best calibration technique, RankMixup [ 32 ], RelCal reduces the ECE by $4.25\%$ 4.25 % on the challenging imbalanced dataset CIFAR-100-LT. Additionally, on the large-scale ImageNet dataset, RelCal reduces the AECE from $9.45\%$ 9.45 % to $3.08\%$ 3.08 % — a $6.37\%$ 6.37 % improvement over the baseline model trained with NLL loss.