Multi-Task Affective Behaviour Analysis Based on MT-EmotiNet Models

Abstract

This paper introduces the novel efficient approach for one of the tasks of the seventh Affective Behavior Analysis in-the-wild (ABAW) competition, namely, multi-task learning (MTL) for simultaneous prediction of facial expression, valence, arousal, and detection of action units. We propose an efficient pipeline based on frame-level facial feature extractors pre-trained in multi-task settings to estimate valence-arousal and basic facial expressions given a facial photo. The lightweight architectures of neural networks trained in multi-task scenarios (MT-EmotiNet) are used, such as MT-EmotiDDAMFNet, MT-EmotiEffNet, and MT-EmotiMobileFaceNet, that can run even on a mobile device without the need to send facial video to a remote server. It was demonstrated that a significant step in improving the overall accuracy is the smoothing of neural network output scores using Gaussian or box filters. It was experimentally demonstrated that such a simple post-processing of predictions from simple blending of two top visual models improves the F1-score of facial expression recognition up to 7%. At the same time, the mean Concordance Correlation Coefficient (CCC) of valence and arousal is increased by up to 1.25 times compared to each model’s frame-level predictions. Our final performance score on the validation set from the multi-task learning challenge is 4.5 times higher than the baseline (1.494 vs 0.32). As a result, we took the second place in this challenge.