A Data-Centric Module for Neural Rendering

Abstract

Recent advances in Novel-View Synthesis (NVS) and 3D Generation (3DGen) from 2D images have marked significant progress in various domains. While the Structure-from-Motion (SfM) and Multi-View Stereo (MVS) pipelines remain prevalent, their limitations have driven the exploration of Deep Learning (DL)-based methods. Among these, Neural Radiance Fields (NeRFs) stand out for their exceptional capabilities in novel view synthesis and 3D reconstruction. However, their reliance on large, diverse 2D images for training, which capture the same scene from different perspectives, poses challenges. To address these challenges, our research proposes a module that introduces innovative data-centric strategies to improve the fidelity of novel view synthesis and reconstruction of NeRFs. In particular, the adopted strategy relies on depth priors, RGB masks, geometrical warping, and deep learning-based image restoration to improve the training and performance of NeRF models, following a human-in-the-loop approach. This module paves the way for a novel data-centric and DL-driven, to improve performances in NeRFs, which is adaptable across different NeRF architectures. Through a comprehensive quantitative-qualitative analysis of such a framework, on a challenging NeRF benchmark dataset, we demonstrate the effectiveness and versatility of our approach.