Spectral Distribution Alignment for Enhanced Generalization in Regression

Abstract

While several techniques have been proposed to enhance the generalization of deep learning models for classification problems, limited research has been conducted on improving generalization for regression tasks. This is primarily due to the continuous nature of regression labels, which makes it challenging to directly apply classification-based techniques to regression tasks. In this paper, we introduce a novel generalization method for regression tasks based on the metric learning assumption that the distance between features and labels should be proportional. Unlike previous approaches that solely consider the prediction of this proportion as constant and disregard its variation among samples, we argue that this proportion can be defined as a mapping function. Additionally, we propose minimizing the error of this function and stabilizing its fluctuating behavior by smoothing out its variations. To further enhance Out-of-Distribution (OOD) generalization, we leverage the characteristics of the spectral norm ( i.e. , the sub-multiplicativity of the spectral norm of the feature matrix can be expressed as Frobenius norm of the output), and align the maximum singular value of the feature matrices across different domains. We conduct experiments on 5 datasets for OOD generalization in regression, and our method consistently outperforms state-of-the-art approaches in the majority of cases. Our code is released at https://github.com/workerbcd/SCR .