Joint Motion Detection in Neural Videos Training

Abstract

Neural radiance fields (NeRF) can produce photo realistic free-viewpoint images. Recently, incremental neural video training approaches took a step towards interactive streaming via a frame-by-frame approach naturally free of lag. Motion detection in neural videos via a frame-byframe approach can provide valuable cues to enable temporally stable neural videos suitable for interactive streaming. In addition, motion cues can be used to guide the ray sampling phase to model dynamic regions more efficiently. Hence, motion detection can be a key component in telepresence/social networking and immersive cloud gaming applications. In this paper, we propose a novel approach that computes static/dynamic separation masks with high accuracy and spatial coherency across different views together with NeRF optimization process. This is enabled by using explicit deformation network instead of implicit motions/structure layers (novel network architecture) as well as novel specifically designed training schedule. To the best of our knowledge, this is the first work that enables motion estimation via a frame-by-frame approach in a neural video training. The proposed work is desirable as it does not require buffer chunks of frames available before processing and hence is suitable for interactive streaming scenarios. Experimental results shows the effectiveness of the proposed motion detection approach in neural videos.