Pose and Expression-Coherent Face Recovery in the Wild

Abstract

We present a novel method to recover images of faces, particularly when large spatial regions of the face are unavailable due to data losses or occlusions. In contrast with previous work, we do not make assumptions on the data neither during training nor testing (such as assuming that the person was seen before or that all faces are perfectly aligned and have identical head pose, expression, etc.). Instead, we propose to tackle the problem in a purely unsupervised way, leveraging a large face dataset. During training, first we cluster faces based on their landmark's positions (obtained by an automatic face landmark estimator). Then, we model the face appearance for each group using sparse coding with learned dictionaries, with one dictionary per cluster. At test time, given a face to recover, we find its belonging cluster and occluded area and restore missing pixels by applying the group-specific sparse appearance representation learned during training. We show results on two "in the wild" datasets. Our method shows promising results on challenging faces and our sparse coding approach outperforms prior subspace learning techniques.