Convergence of an Actor-Critic Gradient Flow for Entropy Regularised MDPs in General Spaces
Abstract
We prove the stability and global convergence of a coupled actor-critic gradient flow for infinite-horizon and entropy-regularised Markov decision processes (MDPs) in continuous state and action space with linear function approximation under Q-function realisability. We consider a version of the actor critic gradient flow where the critic is updated using temporal difference (TD) learning while the policy is updated using a policy mirror descent method on a separate timescale. For general action spaces, the relative entropy regularizer is unbounded and thus it is not clear a priori that the actor-critc flow does not suffer from finite-time blow-up. Therefore we first demonstrate stability which in turn enables us obtain a convergence rate of the actor critic flow to the optimal regularised value function. The arguments presented show that timescale separation is crucial for stability and convergence in this setting.
Cite
Text
Zorba et al. "Convergence of an Actor-Critic Gradient Flow for Entropy Regularised MDPs in General Spaces." International Conference on Learning Representations, 2026.Markdown
[Zorba et al. "Convergence of an Actor-Critic Gradient Flow for Entropy Regularised MDPs in General Spaces." International Conference on Learning Representations, 2026.](https://mlanthology.org/iclr/2026/zorba2026iclr-convergence/)BibTeX
@inproceedings{zorba2026iclr-convergence,
title = {{Convergence of an Actor-Critic Gradient Flow for Entropy Regularised MDPs in General Spaces}},
author = {Zorba, Denis and Siska, David and Szpruch, Lukasz},
booktitle = {International Conference on Learning Representations},
year = {2026},
url = {https://mlanthology.org/iclr/2026/zorba2026iclr-convergence/}
}