CMAD: Correlation-Aware and Modalities-Aware Distillation for Multimodal Sentiment Analysis with Missing Modalities

Abstract

Multimodal Sentiment Analysis (MSA) enhances emotion recognition by integrating information from multiple modalities. However, multimodal learning with missing modalities suffers from representation inconsistency and optimization instability, leading to suboptimal performance. In this paper, we introduce Correlation-Aware and Modalities-Aware Distillation (CMAD), a unified framework designed for MSA under varying missing-modality conditions. Specifically, CMAD comprises two key components: (1) Correlation-Aware Feature Distillation (CAFD), which enforces multi-level representation alignment by preserving both feature similarities and high-order correlation structures between teacher and student models, and (2) Modality-Aware Regularization (MAR) employs an adaptive weighting strategy guided by modality difficulty, enabling a curriculum learning paradigm to stabilize the training process. Extensive evaluations on five datasets show that CMAD consistently outperforms existing methods, achieving average performance improvements of 1.0% on MOSEI, 4.4% on IEMOCAP, 1.9% on MUStARD, 0.5% on UR-FUNNY and 1.9% on CHERMA. Code is available at: https://github.com/YetZzzzzz/CMAD.