Constrained Imitation Q-Learning with Earth Mover’s Distance Reward

Abstract

We propose constrained Earth Mover's Distance (CEMD) Imitation Q-learning that combines exploration of Reinforcement Learning (RL) and the sample efficiency of Imitation Learning (IL). Sample efficiency makes CEMD suitable for robot learning. Immediate rewards can be efficiently computed by a greedy Earth Mover's Distance (EMD) variant between observed state-action pairs and state-actions in the stored expert demonstrations. In CEMD, we constrain the previously proposed non-stationary greedy EMD reward by proposing a greedy EMD upper bound estimate and a generic Q-learning lower bound. In PyBullet continuous control benchmarks, CEMD is more sample efficient, achieves higher performance, and yields less variance than its competitors.