Sparse-View 3D Reconstruction of Clothed Humans via Normal Maps

Abstract

We present a novel deep learning-based approach to the 3D reconstruction of clothed humans using weak supervision via 2D normal maps. Given a single RGB image or multiview images our network is optimized to infer a person-specific signed distance function (SDF) discretized on a tetrahedral mesh surrounding the body in a rest pose. Subsequently estimated human pose and camera parameters are used to generate a normal map from the SDF. A key aspect of our approach is the direct use of the Marching Tetrahedra algorithm in end-to-end optimization and in order to do so we derive analytical gradients to facilitate straightforward differentiation (and thus backpropagation). Additionally predicted normal maps allow us to leverage pretrained image-to-normal networks in order to minimize a surface error instead of a photometric error. We demonstrate the efficacy of our approach on both labeled and in-the-wild data in the context of existing clothed human reconstruction methods.