Infinite Leagues Under the Sea: Realistic 3D Underwater Terrain Generation Augmented by Visual Foundation Models

Abstract

This paper introduces an underwater 3D terrain generation method. Off-the-shelf models trained on internet-scale data exhibit downgraded realism when generating less common scenes, e.g. seafloor. In this work, we resort to the image database collected from underwater robot surveys, which contains massive real seafloor observations and covers large areas. We leverage visual foundation models to extract 3D geometry and semantics from the data, and train a diffusion model that generates realistic seafloor images in RGBD channels, conditioned on latent embeddings. The generated images can be further stitched together into a 3D map, and refined as a 3DGS model which allows photorealistic novel view rendering. We envision that this technology can infinitely generate underwater terrains for potential applications in filming, gaming, and robot simulation.