Extending Layered Models to 3D Motion

Abstract

We consider the problem of inferring a layered representa-tion, its depth ordering and motion segmentation from a video in whichobjects may undergo 3D non-planar motion relative to the camera. Wegeneralize layered inference to the aforementioned case and correspond-ing self-occlusion phenomena. We accomplish this by introducing a flat-tened 3D object representation, which is a compact representation of anobject that contains all visible portions of the object seen in the video,including parts of an object that are self-occluded (as well as occluded)in one frame but seen in another. We formulate the inference of such flat-tened representations and motion segmentation, and derive an optimiza-tion scheme. We also introduce a new depth ordering scheme, which isindependent of layered inference and addresses the case of self-occlusion.It requires almost no computation given the flattened representations.Experiments on benchmark datasets show the advantage of our methodcompared to existing layered methods, which do not model 3D motionand self-occlusion.