Reinforcement Learning in POMDPs with Memoryless Options and Option-Observation Initiation Sets

Denis Steckelmacher, Diederik M. Roijers, Anna Harutyunyan, Peter Vrancx, Hélène Plisnier, Ann Nowé

AAAI 2018 pp. 4099-4106

doi:10.1609/AAAI.V32I1.11606 /aaai/2018/steckelmacher2018aaai-reinforcement/

Abstract

Many real-world reinforcement learning problems have a hierarchical nature, and often exhibit some degree of partial observability. While hierarchy and partial observability are usually tackled separately (for instance by combining recurrent neural networks and options), we show that addressing both problems simultaneously is simpler and more efficient in many cases. More specifically, we make the initiation set of options conditional on the previously-executed option, and show that options with such Option-Observation Initiation Sets (OOIs) are at least as expressive as Finite State Controllers (FSCs), a state-of-the-art approach for learning in POMDPs. OOIs are easy to design based on an intuitive description of the task, lead to explainable policies and keep the top-level and option policies memoryless. Our experiments show that OOIs allow agents to learn optimal policies in challenging POMDPs, while being much more sample-efficient than a recurrent neural network over options.

PDF AAAI Semantic Scholar

Cite

Text

Steckelmacher et al. "Reinforcement Learning in POMDPs with Memoryless Options and Option-Observation Initiation Sets." AAAI Conference on Artificial Intelligence, 2018. doi:10.1609/AAAI.V32I1.11606

Markdown

[Steckelmacher et al. "Reinforcement Learning in POMDPs with Memoryless Options and Option-Observation Initiation Sets." AAAI Conference on Artificial Intelligence, 2018.](https://mlanthology.org/aaai/2018/steckelmacher2018aaai-reinforcement/) doi:10.1609/AAAI.V32I1.11606

BibTeX

@inproceedings{steckelmacher2018aaai-reinforcement,
  title     = {{Reinforcement Learning in POMDPs with Memoryless Options and Option-Observation Initiation Sets}},
  author    = {Steckelmacher, Denis and Roijers, Diederik M. and Harutyunyan, Anna and Vrancx, Peter and Plisnier, Hélène and Nowé, Ann},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year      = {2018},
  pages     = {4099-4106},
  doi       = {10.1609/AAAI.V32I1.11606},
  url       = {https://mlanthology.org/aaai/2018/steckelmacher2018aaai-reinforcement/}
}