Improving 3D Face Details Based on Normal mAP of Hetero-Source Images

Abstract

For each person, there exist large unstructured photo collections in personal photo albums. We call these photos Hetero-source images, which imply abundant shape and texture information of the specific face. In this paper, we propose a novel 3D face modeling method combining the normal map of Hetero-source images with the fitting result based on a single image to achieve more accurate 3D shape estimates. Based on recent research showing that the set of images of convex Lambertian surfaces under general illumination can be well approximated using low-order spherical harmonics, we first incorporate spherical harmonics into the 3D morphable model to initialize the 3D shape. The fitting result, however, suffers from model dominance and lacks of fine details. The normal map inferred by Hetero-source image shading constraints allows the possibility of improving local details and challenging the model dominance. We estimate the normal map which contains more accurate orientation information in an alternating optimization way and apply it to improve the preliminary 3D surface. Experimental results on both synthetic and real world data demonstrate that our method could be used to capture discriminating facial features and outperforms the single image fitting result in accuracy.