Stochastic Rigidity: Image Registration for Nowhere-Static Scenes

Abstract

We consider the registration of sequences of images where the observed scene is entirely non-rigid for example a camera flying over water, a panning shot of a field of sunflowers in the wind, or footage of a crowd applauding at a sports event. In these cases, it is not possible to impose the constraint that world points have similar colour in successive views, so existing registration techniques cannot be applied. Indeed the relationship between a point's colours in successive frames is essentially a random process. However by treating the sequence of images as a set of samples from a multidimensional stochastic time-series, we can learn a stochastic model (e.g. an AR model) of the random process which generated the sequence of images. With a static camera, this stochastic model can be used to extend the sequence arbitrarily in time. Driving the model with random noise results in an infinitely varying sequence of images which always looks like the short input sequence. In this way, we can create "videotextures" which can play forever without repetition. With a moving camera, the image generation process comprises two components-a stochastic component-generated by the videotexture, and a parametric component due to the camera motion.