Why Is Posterior Sampling Better than Optimism for Reinforcement Learning?
Abstract
Computational results demonstrate that posterior sampling for reinforcement learning (PSRL) dramatically outperforms existing algorithms driven by optimism, such as UCRL2. We provide insight into the extent of this performance boost and the phenomenon that drives it. We leverage this insight to establish an $\tilde{O}(H\sqrt{SAT})$ Bayesian regret bound for PSRL in finite-horizon episodic Markov decision processes. This improves upon the best previous Bayesian regret bound of $\tilde{O}(H S \sqrt{AT})$ for any reinforcement learning algorithm. Our theoretical results are supported by extensive empirical evaluation.
Cite
Text
Osband and Van Roy. "Why Is Posterior Sampling Better than Optimism for Reinforcement Learning?." International Conference on Machine Learning, 2017.Markdown
[Osband and Van Roy. "Why Is Posterior Sampling Better than Optimism for Reinforcement Learning?." International Conference on Machine Learning, 2017.](https://mlanthology.org/icml/2017/osband2017icml-posterior/)BibTeX
@inproceedings{osband2017icml-posterior,
title = {{Why Is Posterior Sampling Better than Optimism for Reinforcement Learning?}},
author = {Osband, Ian and Van Roy, Benjamin},
booktitle = {International Conference on Machine Learning},
year = {2017},
pages = {2701-2710},
volume = {70},
url = {https://mlanthology.org/icml/2017/osband2017icml-posterior/}
}