Zero-Shot Audio-Visual Compound Expression Recognition Method Based on Emotion Probability Fusion
Abstract
A Compound Expression Recognition (CER) as a sub-field of affective computing is a novel task in intelligent human-computer interaction and multimodal user interfaces. We propose a novel audio-visual method for CER. Our method relies on emotion recognition models that fuse modalities at the emotion probability level, while decisions regarding the prediction of compound expressions are based on the pair-wise sum of weighted emotion probability distributions. Notably, our method does not use any training data specific to the target task. Thus, the problem is a zero-shot classification task. The method is evaluated in multi-corpus training and cross-corpus validation setups. We achieved F1 scores of 32.15% and 25.56% for the AffWild2 and C-EXPR-DB test subsets without training on target corpus and target task, respectively. Therefore, our method is on par with methods developed training target corpus or target task. The source code is publicly available from https://elenaryumina.github.io/AVCER/.
Cite
Text
Ryumina et al. "Zero-Shot Audio-Visual Compound Expression Recognition Method Based on Emotion Probability Fusion." IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2024. doi:10.1109/CVPRW63382.2024.00478Markdown
[Ryumina et al. "Zero-Shot Audio-Visual Compound Expression Recognition Method Based on Emotion Probability Fusion." IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2024.](https://mlanthology.org/cvprw/2024/ryumina2024cvprw-zeroshot/) doi:10.1109/CVPRW63382.2024.00478BibTeX
@inproceedings{ryumina2024cvprw-zeroshot,
title = {{Zero-Shot Audio-Visual Compound Expression Recognition Method Based on Emotion Probability Fusion}},
author = {Ryumina, Elena and Markitantov, Maxim and Ryumin, Dmitry and Kaya, Heysem and Karpov, Alexey},
booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops},
year = {2024},
pages = {4752-4760},
doi = {10.1109/CVPRW63382.2024.00478},
url = {https://mlanthology.org/cvprw/2024/ryumina2024cvprw-zeroshot/}
}