Reinforcement Learning with Time

Abstract

This paper steps back from the standard infinite hori-zon formulation of reinforcement learning problems to consider the simpler case of finite horizon problems. Although finite horizon problems may be solved us-ing infinite horizon learning algorithms by recasting the problem as an infinite horizon problem over a state space extended to include time, we show that such an application of infinite horizon learning algorithms does not make use of what is known about the environment structure, and is therefore inefficient. Preserving a no-tion of time within the environment allows us to con-sider extending the environment model to include, for example, random action duration. Such extentions al-low us to model non-Markov environments which can be learned using reinforcement learning algorithms.