Qualitative Control Learning Can Be Much Faster than Reinforcement Learning

Abstract

Reinforcement learning has emerged as a prominent method for controlling dynamic systems in the absence of a precise mathematical model. However, its reliance on extensive interactions with the environment often leads to prolonged training periods. In this paper, we propose an alternative approach to learning control policies that focuses on learning qualitative models and uses symbolic planning to derive a qualitative plan for the control task, which is executed by an adaptive reactive controller. We conduct experiments utilizing our approach on the cart-pole problem, a standard benchmark in dynamic system control. We additionally extend this problem domain to include uneven terrains, such as driving over craters or hills, to assess the robustness of learned controllers. Our results indicate that qualitative learning offers significant advantages over reinforcement learning in terms of sample efficiency, transferability, and interpretability. We demonstrate that our proposed approach is at least two orders of magnitude more sample efficient in the cart-pole domain than the usual variants of reinforcement learning.