Actual Return Reinforcement Learning Versus Temporal Differences: Some Theoretical and Experimental Results

Abstract

This paper argues that for many domains, we can expect credit-assignment methods that use actual returns to be more effective for reinforcement learning than the more commonly used temporal difference methods. We present analysis and empirical evidence from three sets of experiments in different domains to support this claim. A new algorithm we call C-Trace, a variant of the P-Trace RL algorithm is introduced, and some possible advantages of using algorithms of this type are discussed. 1 Introduction Whether it is better to learn on the basis of actual outcomes (also known as rollouts or actual returns), as in Monte Carlo methods, or to learn on the basis of interim estimates, as in temporal difference (TD) methods, has been identified as a key question in reinforcement learning (RL) [17]. The evidence to date has been mixed. In [17], Sutton points out the former enjoy a number of theoretical advantages when certain classes of function approximators are used [7, 3]; Boyan and Moore [...