Addressing Task Conflicts in LLMs Multi-Task Fine-Tuning with Task-Specific Subnetwork Refinement

Abstract

Instruction tuning is a widely-adopted technique for adapting Large Language Models (LLMs) to specific domains such as coding and mathematics. However, when instruction tuning is performed on multiple tasks simultaneously, LLMs often experience performance degradation due to conflicts between different tasks. In this work, we explore how task conflicts arise by analyzing the subnetworks within the model for accomplishing different tasks. Our analysis reveals that these subnetworks operate largely independently with minimal overlap. The findings indicate that task conflicts occur because traditional training methods update all model parameters simultaneously for all tasks, causing gradients from one task to disrupt the specialized subnetworks of others. To address this issue, we propose Task-Specific Subnetwork Refinement (TSSR), a novel training framework that mitigates task conflicts by selectively refining the subnetworks associated with each task. Our approach first locates these task-specific subnetworks and then applies gradients exclusively to the relevant subnetwork during training, thereby reducing interference between tasks. Experimental results show that TSSR significantly improves model performance across multiple tasks, including coding, translation, and reasoning tasks. Furthermore, training efficiency is enhanced by reducing the computational burden of gradient updates.