VRS-NeRF: Visual Relocalization with Sparse Neural Radiance Field

Abstract

Visual localization is a key technique to various applications including autonomous driving, robotics, and AR/VR. After many years of exploration, most previous classic methods are either inefficient or inaccurate. In this paper, we leverage the powerful scene compressing ability of neural radiance fields (NeRFs) for map representation and propose an efficient and accurate framework, called VRS-NeRF, for visual localization with sparse neural radiance field. To be more specific, we introduce an explicit geometric map (EGM) for 3D map representation and an implicit learned map (ILM) for sparse patches rendering. In this localization process, EGP provides priors of spare 2D points and ILM utilizes these sparse points to render patches with sparse NeRFs for matching. This allows us to discard a large number of 2D descriptors so as to reduce the map size. Moreover, rendering patches only for useful points rather than all pixels in the whole image reduces the rendering time significantly. This framework inherits the accuracy of hierarchical methods (HMs) and discards their low efficiency. Experiments on 7Scenes, CambridgeLandmarks, and Aachen datasets show that our method gives much better accuracy than absolute pose regression (APR) and scene-coordinate regression (SCR), and also reports close performance to HMs but is much more efficient. Code is available at https://github.com/feixue94/vrs-nerf .