Mono Camera Multi-View Diminished Reality

Abstract

Diminished reality (DR) aims at removing objects auto-matically in live video streams, in a way which is unrecognizable to human observers. Approaches working on mobile devices utilize image inpainting algorithms to fill the holes (target region) of the removed objects in a plausible way. We propose a new DR algorithm, which extracts a scene model from a sparse point cloud of a mono SLAM algorithm. The scene model allows to inpaint complex scenes consisting of multiple planes intersecting at the target region. A plane-wise key frame estimation preserves scene structure as well as improving inpainting results. Furthermore, we introduce a viewing angle-based key frame update policy, ensuring high quality of the key frames. The multi-threaded design of the algorithm achieves nearly real-time performance on a mobile device.