Multi-Angle Consistent Generative NeRF with Additive Angular Margin Momentum Contrastive Learning

Abstract

The NeRF and GAN-based GIRAFFE algorithm has drawn a lot of attention because of its controllable image production capacity. However, the consistency of GIRAFFE rendering results from different perspectives of the same object is not stable. The reasons are twofold: First, the optimization goal of GIRAFFE is only concerned with whether the generated image resembles the real image or not. Second, GIRAFFE could learn knowledge implicitly to complement the feature deformation of large camera angle change, which may introduce uncontrollable generation mode resulting in low consistency of the 3D object. This limits its application in fields such as digital person generation and biometric identity. In this paper, We introduce an additional Encoder to form a momentum-based Contrastive Learning with the Discriminator of GAN. In addition, we propose an AamNCE loss to train our model which introduces an additive angular margin to the positive sample pairs. In brief, the proposed framework could be regarded as a new paradigm of GAN and Contrastive Learning. The Contrastive Learning improves the characteristic expression ability of the model, and the AamNCE loss makes the category boundaries of the generated images more explicit. The experimental results demonstrate that our method maintains the consistency of face identity well in the multi-angle rotation of the face dataset.