Efficient Memory-Based Dynamic Programming

Abstract

A novel memory-based approach to dynamic programming that addresses the issue of generalization is presented. In this approach action values are represented by storing actual experiences in a memory and computed by a kind of locally weighted regression, and generalizations are made by searching the memory for relevant experience. The new approach does not require the quantization of continuous state or action spaces and can achieve arbitrarily variable resolution. By concentrating on important areas of the state space while ignoring the rest, the method represents an attempt to dodge Bellman's curse of dimensionality. This memory-based dynamic programming method has been implemented on a parallel machine, the Connection Machine, and used to successfully model and control a cart-pole system.