Scaling Reinforcement Learning Techniques via Modularity

Abstract

Complex tasks can often be decomposed into subgoals or chunks that can be solved individually. This chapter describes how reinforcement learning techniques can take advantage of this. By decomposing a reinforcement learning architecture into modules, where each module learns to solve a subgoal of the task, both structural and temporal credit transfer can be improved. The chapter also presents a variation of Q-learning that allows the modular architecture to reduce the effects of perceptual aliasing on reward estimation. Q-learning defines a mechanism for propagating estimates of expected utility backwards to the previous actions that resulted in this utility. However, it does not define a mechanism that allows the system to generalize, that is, to allow experience gained for one state to transfer to similar states. This is a serious problem because in naive representations the number of possible states often grows exponentially with the number of bits to be represented. Several approaches have been invented to deal with this scaling problem. White head and Ballard have used indexical representations to avoid having to estimate a Q-value for each distinct world state and each action's possible variable binding. For more complicated tasks that involve generating sequences of actions, a robot's state vector might be devoted mostly to representing internal state, and only a small fraction used to represent perceptual data. In such situations, state vectors that seem very similar may represent very different situations and should not share estimates.