Discrepancy-Guided Parameter Suppression for Robust Fine-Tuning

Abstract

Foundation models (FMs) have demonstrated remarkable success in zero-shot learning and transferability across a broad range of unseen tasks. However, despite their robustness, fine-tuning these models on specific downstream tasks often leads to a trade-off: improvements in in-distribution (ID) performance typically come at the expense of out-of-distribution (OOD) generalization. To address this, recent research has focused on strategies that balance performance on the target dataset while retaining robustness on unseen data. In this paper, we propose a novel fine-tuning method that leverages parameter discrepancy between pre-trained and fine-tuned models to identify ID-specific parameters prone to overfitting. Our hypothesis is that parameters undergoing the most significant changes during fine-tuning are more likely to capture task-specific information. We introduce a Discrepancy-guided Parameter Suppression (DPS) mechanism to rank parameters with discrepancy score and selectively suppress those with the highest discrepancies to prevent overfitting. This approach encourages the model to learn task-invariant representations, improving OOD generalization. We evaluate our method on the DomainNet image classification benchmark, achieving a 1\% improvement in OOD performance over the state-of-the-art method, without sacrificing ID performance. Additionally, we analyze the effects of parameter suppression percentages, selection granularity, and normalization strategies on discrepancy scores, providing comprehensive insights into robust fine-tuning.