Improving Valence-Arousal Estimation with Spatiotemporal Relationship Learning and Multimodal Fusion

Jun Yu, Gongpeng Zhao, Yongqi Wang, Zhihong Wei, Zerui Zhang, Zhongpeng Cai, Guochen Xie, Jichao Zhu, Wangyuan Zhu, Shuoping Yang, Yang Zheng, Qingsong Liu, Jiaen Liang

CVPRW 2024 pp. 7878-7885

doi:10.1109/CVPRW63382.2024.00785 /cvprw/2024/yu2024cvprw-improving/

Abstract

This paper presents our approach for the VA (Valence-Arousal) estimation task in the ABAW6 competition. We devised a comprehensive model by preprocessing video frames and audio segments to extract visual and audio features. Through the utilization of Temporal Convolutional Network (TCN) module, we effectively captured the temporal and spatial correlations between these features. Subsequently, we employed a Transformer encoder structure to learn long-range dependencies, thereby enhancing the model’s performance and generalization ability. Additionally, we proposed the LA-SE module to better capture local image information and enhance channel selection and suppression. Our method leverages a multimodal data fusion approach, integrating pre-trained audio and video backbones for feature extraction, followed by TCN-based spatiotemporal encoding and Transformer-based temporal information capture. Experimental results demonstrate the effectiveness of our approach, achieving competitive performance in VA estimation on the AffWild2 dataset.

CVPRW Semantic Scholar

Cite

Text

Yu et al. "Improving Valence-Arousal Estimation with Spatiotemporal Relationship Learning and Multimodal Fusion." IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2024. doi:10.1109/CVPRW63382.2024.00785

Markdown

[Yu et al. "Improving Valence-Arousal Estimation with Spatiotemporal Relationship Learning and Multimodal Fusion." IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2024.](https://mlanthology.org/cvprw/2024/yu2024cvprw-improving/) doi:10.1109/CVPRW63382.2024.00785

BibTeX

@inproceedings{yu2024cvprw-improving,
  title     = {{Improving Valence-Arousal Estimation with Spatiotemporal Relationship Learning and Multimodal Fusion}},
  author    = {Yu, Jun and Zhao, Gongpeng and Wang, Yongqi and Wei, Zhihong and Zhang, Zerui and Cai, Zhongpeng and Xie, Guochen and Zhu, Jichao and Zhu, Wangyuan and Yang, Shuoping and Zheng, Yang and Liu, Qingsong and Liang, Jiaen},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops},
  year      = {2024},
  pages     = {7878-7885},
  doi       = {10.1109/CVPRW63382.2024.00785},
  url       = {https://mlanthology.org/cvprw/2024/yu2024cvprw-improving/}
}