Planning from Point Clouds over Continuous Actions for Multi-Object Rearrangement

Abstract

Multi-object rearrangement is a challenging task that requires robots to reason about a physical 3D scene and the effects of a sequence of actions. While traditional task planning methods are shown to be effective for long-horizon manipulation, they require discretizing the continuous state and action space into symbolic descriptions of objects, object relationships, and actions. Our proposed method is instead able to take in a partially-observed point cloud observation of an initial scene and plan to a goal-satisfying configuration, without needing to discretize the set of actions or object relationships. To enable this, we formulate the planning problem as an A* search over the space of possible point cloud rearrangements. We sample point cloud transformations from a learned, domain-specific prior and then search for a sequence of such point cloud transformations that leads from the initial state to a goal. We evaluate our method in terms of task planning success and task execution success on a real-world, multi-step table bussing environment and a simulation block stacking environment. We experimentally demonstrate that our method produces successful plans and outperforms a policy-learning approach; we also perform ablations that show the importance of search in our approach.