The Dynamics of Reinforcement Learning in Cooperative Multiagent Systems

Abstract

Reinforcement learning can provide a robust and natural means for agents to learn how to coordinate their action choices in multiagent systems. We examine some of the factors that can influence the dynamics of the learning process in such a setting. We first distinguish reinforcement learners that are unaware of (or ignore) the presence of other agents from those that explicitly attempt to learn the value of joint actions and the strategies of their counterparts. We study Q-learning in cooperative multiagent systems under these two perspectives, focusing on the influence of partial action observability, game structure, and exploration strategies on convergence to (optimal and suboptimal) Nash equilibria and on learned Qvalues. 1 Introduction The application of learning to the problem of coordination in multiagent systems (MASs) has become increasingly popular in AI and game theory. The use of reinforcement learning (RL), in particular, has attracted recent attention [22, 17, 16, 13, ...