Learning Intuitive Physics by Explaining Surprise

Abstract

The IntPhys Challenge aims to evaluate how well algorithms capture "common sense" about the physical world by measuring the ability to detect violations of intuitive physics in dynamic multi-object visual scenes. One approach to this problem is to define or learn a detailed model of the observations and dynamics and to then detect violations of that model. While viable, this approach poses challenges in acquiring an accurate enough model that can handle detailed non-linear object interactions, such as visual occlusion and collisions. In this work, we consider an alternative approach, the Surprise and Explain (SnE) framework, which aims for simplicity while remaining highly flexible. The key idea is to exploit the assumption that, for the vast majority of time, objects follow simple dynamic models, e.g. linear dynamics. Further, when the simple dynamics are occasionally violated ("surprises") due to non-linear interactions, e.g. collisions and occlusion, it is assumed that there is a small set of detectable explanations for the surprise. Violations of intuitive physics then correspond to surprises for which an explanation cannot be inferred. This paper develops an instantiation of the SnE framework and demonstrates its potential in the IntPhys Challenge by placing 2nd at the time of this paper’s submission.1