A Convergent Reinforcement Learning Algorithm in the Continuous Case: The Finite-Element Reinforcement Learning

Abstract

This paper presents a direct reinforcement learning algorithm, called Finite Element Reinforcement Learning, in the continuous case. We propose a continuous formalism for the studying of reinforcement learning using the continuous optimal framework, then we state the associated Hamilton-Jacobi-Bellman equation. First, we propose to approximate the value function by a numerical scheme based on a finite-element method. This generates a discrete Markov decision process, with finite state and control spaces, which can be solved by dynamic programming. The computation of this approximation scheme, in reinforcement learning terminology, belongs to the class of indirect learning methods. Then we present our direct learning algorithm which approximates the previous finite-element scheme and prove its convergence to the value function of the continuous problem. / Cet article présente un algorithme d'apprentissage par renforcement, appelé "Finite Element Reinforcement Learning" dans le cas continu. Cet algorithme est basé sur des méthodes aux éléments finis.