Jensen-Tsallis Divergence for Supervised Classification Under Data Imbalance

Abstract

Abstract In supervised classification problems using Deep Neural Networks, the loss function is typically based on the Kullback–Leibler divergence. However, alternative entropic divergence formulations, such as the Jensen–Shannon Divergence (JSD), have recently garnered attention for their unique properties. In this study, we delve deeper into the interpretation of the JSD and its generalized form, the Jensen–Tsallis Divergence (JTD), as alternative loss functions for supervised classification. When provided with one-hot encoded distributions for the true label probabilities, we demonstrate that these novel divergences impose an intrinsic output confidence regularization that prevents overfitting. Additionally, the q non-extensive parameter of the JTD directly influences the structure of the regularizer, offering increased flexibility in the formulation of the loss function. Through experiments conducted on artificially imbalanced versions of MNIST, Fashion-MNIST, SVHN and CIFAR-10 we showcase how JTD outperforms JSD and other traditional loss functions in terms of generalization performance, especially for highly imbalanced datasets.