Sparse Kernel-SARSA(λ) with an Eligibility Trace
Abstract
We introduce the first online kernelized version of SARSA( λ ) to permit sparsification for arbitrary λ for 0 ≤ λ ≤ 1; this is possible via a novel kernelization of the eligibility trace that is maintained separately from the kernelized value function. This separation is crucial for preserving the functional structure of the eligibility trace when using sparse kernel projection techniques that are essential for memory efficiency and capacity control. The result is a simple and practical Kernel-SARSA( λ ) algorithm for general 0 ≤ λ ≤ 1 that is memory-efficient in comparison to standard SARSA( λ ) (using various basis functions) on a range of domains including a real robotics task running on a Willow Garage PR2 robot.
Cite
Text
Robards et al. "Sparse Kernel-SARSA(λ) with an Eligibility Trace." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2011. doi:10.1007/978-3-642-23808-6_1Markdown
[Robards et al. "Sparse Kernel-SARSA(λ) with an Eligibility Trace." European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, 2011.](https://mlanthology.org/ecmlpkdd/2011/robards2011ecmlpkdd-sparse/) doi:10.1007/978-3-642-23808-6_1BibTeX
@inproceedings{robards2011ecmlpkdd-sparse,
title = {{Sparse Kernel-SARSA(λ) with an Eligibility Trace}},
author = {Robards, Matthew W. and Sunehag, Peter and Sanner, Scott and Marthi, Bhaskara},
booktitle = {European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases},
year = {2011},
pages = {1-17},
doi = {10.1007/978-3-642-23808-6_1},
url = {https://mlanthology.org/ecmlpkdd/2011/robards2011ecmlpkdd-sparse/}
}