On the Heterogeneity of Independent Learning Dynamics in Zero-Sum Stochastic Games
Abstract
We analyze the convergence properties of the two-timescale fictitious play combining the classical fictitious play with the Q-learning for two-player zero-sum stochastic games with player-dependent learning rates. We show its almost sure convergence under the standard assumptions in two-timescale stochastic approximation methods when the discount factor is less than the product of the ratios of player-dependent step sizes. To this end, we formulate a novel Lyapunov function formulation and present a one-sided asynchronous convergence result.
Cite
Text
Sayin and Cetiner. "On the Heterogeneity of Independent Learning Dynamics in Zero-Sum Stochastic Games." Proceedings of The 4th Annual Learning for Dynamics and Control Conference, 2022.Markdown
[Sayin and Cetiner. "On the Heterogeneity of Independent Learning Dynamics in Zero-Sum Stochastic Games." Proceedings of The 4th Annual Learning for Dynamics and Control Conference, 2022.](https://mlanthology.org/l4dc/2022/sayin2022l4dc-heterogeneity/)BibTeX
@inproceedings{sayin2022l4dc-heterogeneity,
title = {{On the Heterogeneity of Independent Learning Dynamics in Zero-Sum Stochastic Games}},
author = {Sayin, Muhammed and Cetiner, Kemal},
booktitle = {Proceedings of The 4th Annual Learning for Dynamics and Control Conference},
year = {2022},
pages = {994-1005},
volume = {168},
url = {https://mlanthology.org/l4dc/2022/sayin2022l4dc-heterogeneity/}
}