Reinforcement Algorithms Using Functional Approximation for Generalization and Their Application to Cart Centering and Fractal Compression

Abstract

We address the conflict between identification and control or alternatively, the conflict be-tween exploration and exploitation, within the framework of reinforcement learning. Q-learning has recently become a popular off-policy reinforcement learning method. The conflict between exploration and exploitation slows down Q-learning algorithms; their per-formance does not scale up and degrades rap-idly as the number of states and actions in-creases. One reason for this slowness is that exploration lacks the ability to extrapolate and interpolate from learning and to a large extent has to "reinvent the wheel". Moreover, not all reinforcement problems one encounters are f i-nite state and action systems. Our approach to solving continuous state and action problems is to approximate the continuous state and ac-tion spaces with finite sets of states and ac-tions and then to apply a finite state and action learning method. This approach provides the means for solving continuous state and action problems but does not yet address the per-formance problem associated with scaling up states and actions. We address the scaling problem using functional approximation methods. Towards that end, this paper intro-duces two new reinforcement algorithms, QLVQ and Quad-Q-learning, respectively, and shows their successful application for cart centering and fractal compression. 1