The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimation
Abstract
We study the problem of temporal-difference-based policy evaluation in reinforcement learning. In particular, we analyse the use of a distributional reinforcement learning algorithm, quantile temporal-difference learning (QTD), for this task. We reach the surprising conclusion that even if a practitioner has no interest in the return distribution beyond the mean, QTD (which learns predictions about the full distribution of returns) may offer performance superior to approaches such as classical TD learning, which predict only the mean return, even in the tabular setting.
Cite
Text
Rowland et al. "The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimation." International Conference on Machine Learning, 2023.Markdown
[Rowland et al. "The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimation." International Conference on Machine Learning, 2023.](https://mlanthology.org/icml/2023/rowland2023icml-statistical/)BibTeX
@inproceedings{rowland2023icml-statistical,
title = {{The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimation}},
author = {Rowland, Mark and Tang, Yunhao and Lyle, Clare and Munos, Remi and Bellemare, Marc G and Dabney, Will},
booktitle = {International Conference on Machine Learning},
year = {2023},
pages = {29210-29231},
volume = {202},
url = {https://mlanthology.org/icml/2023/rowland2023icml-statistical/}
}