Classification of Facial Micro-Expressions Using Motion Magnified Emotion Avatar Images

Abstract

Facial micro-expressions are subtle involuntary movements of the facial muscles, characterized by a rapid, short duration and genuine emotions. The detection and classification of these micro-expressions by humans and machines is challenging due to their short duration and subtlety. These micro-expressions have many important applications, especially in therapy, monitoring and depression analysis. It has been shown that during therapy, the facial micro-expressions of patients diagnosed with depression are very difficult to identify and in most cases are very subtle. In this paper, the primary focus is on recognition of facial micro-expressions and to overcome the class imbalance of the datasets. Firstly, a novel approach that uses multiple magnified ratios of Eulerian motion magnification is applied to the videos to extract the suppressed micro-expressions. Secondly, we remove the micro-expression frames with low textural variance and obtain the Emotion Avatar Image (EAI). Finally, Deep Convolutional Neural Network (CNN) is used to extract robust facial features from the motion magnified EAI images. These features are classified into three different classes: positive, negative and surprise. The approach is evaluated on three spontaneous micro-expression datasets SMIC, SAMM, and CASME II, and the results are compared with the current approaches that show the effectiveness and significance of the approach.