StructureFromGAN: Single Image 3D Model Reconstruction and Photorealistic Texturing

Abstract

We present a generative adversarial model for single photo 3D reconstruction and high resolution texturing. Our framework leverages a neural renderer and a 3D Morphable model of an object. We train our generator on the semantic labelling-to-image translation task. This allows our model to learn rich priors about object appearance and perform all-around texture and shape reconstruction from a single image. Our new generator architecture leverages a power of StyleGAN2 model for image-to-image translation with fine texture detail at the $1024 \times 1024$ 1024 × 1024 resolution. We evaluate our framework quantitatively and qualitatively on Florence Face and Appolo Cars datasets on the tasks of car 3D reconstruction and texturing. Extensive experiments demonstrate that our framework achieves and surpasses the state-of-the-art in single photo 3D object reconstruction and texturing using 3D morphable models. We made our code publicly available ( http://www.zefirus.org/StructureFromGAN ).