EVAEF: Ensemble Valence-Arousal Estimation Framework in the Wild

Abstract

This paper presents our work to the Valence-Arousal Estimation Challenge of the 5th Affective Behavior Analysis in-the-wild (ABAW) competition. We explore the problems in this VA challenge from three aspects: 1) To obtain efficient and robust feature representations, we explore the role of multiple visual and video feature extractors; 2) Based on multimodal feature representations that fuse the visual and video information, we utilize four types of temporal encoders to capture the temporal context information in the video, including the LSTM, GRU, Transformer based encoder and a combined encoder of Transformer and LSTM; 3) five model ensemble strategies are used to combine multiple results with different model settings. Our system achieves the performance in Concordance Correlation Coefficients (CCC) of 0.6193 for valence, 0.6634 for arousal, and a mean CCC of 0.6414 on the test set, which demonstrates the effectiveness of our proposed method and ranks first place in the challenge.