Variance Double-Down: The Small Batch Size Anomaly in Multistep Deep Reinforcement Learning
Abstract
In deep reinforcement learning, multi-step learning is almost unavoidable to achieve state-of-the-art performance. However, the increased variance that multistep learning brings makes it difficult to increase the update horizon beyond relatively small numbers. In this paper, we report the counterintuitive finding that decreasing the batch size parameter improves the performance of many standard deep RL agents that use multi-step learning. It is well-known that gradient variance decreases with increasing batch sizes, so obtaining improved performance by increasing variance on two fronts is a rather surprising finding. We conduct a broad set of experiments to better understand what we call the variance doubledown phenomenon.
Cite
Text
Ceron et al. "Variance Double-Down: The Small Batch Size Anomaly in Multistep Deep Reinforcement Learning." NeurIPS 2022 Workshops: DeepRL, 2022.Markdown
[Ceron et al. "Variance Double-Down: The Small Batch Size Anomaly in Multistep Deep Reinforcement Learning." NeurIPS 2022 Workshops: DeepRL, 2022.](https://mlanthology.org/neuripsw/2022/ceron2022neuripsw-variance/)BibTeX
@inproceedings{ceron2022neuripsw-variance,
title = {{Variance Double-Down: The Small Batch Size Anomaly in Multistep Deep Reinforcement Learning}},
author = {Ceron, Johan Samir Obando and Bellemare, Marc G and Castro, Pablo Samuel},
booktitle = {NeurIPS 2022 Workshops: DeepRL},
year = {2022},
url = {https://mlanthology.org/neuripsw/2022/ceron2022neuripsw-variance/}
}