Bellman Error Based Feature Generation Using Random Projections on Sparse Spaces
Abstract
This paper addresses the problem of automatic generation of features for value function approximation in reinforcement learning. Bellman Error Basis Functions (BEBFs) have been shown to improve the error of policy evaluation with function approximation, with a convergence rate similar to that of value iteration. We propose a simple, fast and robust algorithm based on random projections, which generates BEBFs for sparse feature spaces. We provide a finite sample analysis of the proposed method, and prove that projections logarithmic in the dimension of the original space guarantee a contraction in the error. Empirical results demonstrate the strength of this method in domains in which choosing a good state representation is challenging.
Cite
Text
Fard et al. "Bellman Error Based Feature Generation Using Random Projections on Sparse Spaces." Neural Information Processing Systems, 2013.Markdown
[Fard et al. "Bellman Error Based Feature Generation Using Random Projections on Sparse Spaces." Neural Information Processing Systems, 2013.](https://mlanthology.org/neurips/2013/fard2013neurips-bellman/)BibTeX
@inproceedings{fard2013neurips-bellman,
title = {{Bellman Error Based Feature Generation Using Random Projections on Sparse Spaces}},
author = {Fard, Mahdi Milani and Grinberg, Yuri and Farahmand, Amir-massoud and Pineau, Joelle and Precup, Doina},
booktitle = {Neural Information Processing Systems},
year = {2013},
pages = {3030-3038},
url = {https://mlanthology.org/neurips/2013/fard2013neurips-bellman/}
}