Ray Markov Random Fields for Image-Based 3D Modeling: Model and Efficient Inference

Abstract

In this paper, we present an approach to multi-view image-based 3D reconstruction by statistically inversing the ray-tracing based image generation process. The proposed algorithm is fast, accurate and does not need any initialization. The geometric representation is a discrete volume divided into voxels, with each voxel associated with two properties: opacity (shape) and color (appearance). The problem is then formulated as inferring each voxel's most probable opacity and color through MAP estimation of the developed Ray Markov Random Fields (RayMRF). RayMRF is constructed with three kinds of cliques: the usual unary and pairwise cliques favoring connected voxel regions, and most importantly ray-cliques modelling the ray-tracing based image generation process. Each ray-clique connects the voxels that the viewing ray passes through. It provides a principled way of modeling the occlusion without approximation. The inference problem involved in the MAP estimation is handled by an optimized belief propagation algorithm. One unusual structure of the proposed MRF is that each ray-clique usually involves hundreds/thousands of random variables, which seems to make the inference computationally formidable. Thanks to the special property of the ray-clique functional form, we investigate the deep factorization property of ray-clique energy and get a highly efficient algorithm based on the general loopy belief propagation, which has reduced the computational complexity from exponential to linear. Both of the efficient inference algorithm and the overall system concept are new. Combining these results in an algorithm that can reverse the image generation process very fast. 3D surface reconstruction in a 100×100×100, i.e., 10 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">6</sup> voxel space with 10 images requires roughly 3 minutes on a 3.0 GHz single-core CPU. The running time grows linearly with respect to the number of voxels and the number of images. And the speed could be further improved with a hierarchical sparse representation of the volume, like octree. Experiments on several standard datasets show the quality and speed of the proposed models and algorithms.