Using Transitional Proximity for Faster Reinforcement Learning

Abstract

Why does reinforcement learning take so long? One major reason is that reward spreads too slowly through the agent's policy. When an agent receives reward, existent methods only pass the reward back to internal states along the current path. Normally there are many possible paths to goal states however, and the agent must follow each of them successfully one or more times in order to complete learning. Our algorithm learns the transitions between internal states so that rewards may be passed not only along the one path taken this trial, but also passed back through all transitions learned during previous trials. States closer to the current state receive correspondingly more of the current reward. We call this distance between states in transition space transitional proximity. We explain the basics of reinforcement learning, Q-learning and Kohonen Networks, and then formally develop Transitional Proximity Q-learning in this framework. Experimental results confirm faster learning and much quicker convergence to optimal policies.