CTD4 - A Deep Continuous Distributional Actor-Critic Agent with a Kalman Fusion of Multiple Critics
Abstract
Categorical Distributional Reinforcement Learning (CDRL) has demonstrated superior sample efficiency in learning complex tasks compared to conventional Reinforcement Learning (RL) approaches. However, the practical application of CDRL is encumbered by challenging projection steps, detailed parameter tuning, and domain knowledge. This paper addresses these challenges by introducing a pioneering Continuous Distributional Model-Free RL algorithm tailored for continuous action spaces. The proposed algorithm simplifies the implementation of distributional RL, adopting an actor-critic architecture wherein the critic outputs a continuous probability distribution. Additionally, we propose an ensemble of multiple critics fused through a Kalman fusion mechanism to mitigate overestimation bias. Through a series of experiments, we validate that our proposed method provides a sample-efficient solution for executing complex continuous-control tasks.
Cite
Text
Valencia et al. "CTD4 - A Deep Continuous Distributional Actor-Critic Agent with a Kalman Fusion of Multiple Critics." AAAI Conference on Artificial Intelligence, 2025. doi:10.1609/AAAI.V39I20.35391Markdown
[Valencia et al. "CTD4 - A Deep Continuous Distributional Actor-Critic Agent with a Kalman Fusion of Multiple Critics." AAAI Conference on Artificial Intelligence, 2025.](https://mlanthology.org/aaai/2025/valencia2025aaai-ctd/) doi:10.1609/AAAI.V39I20.35391BibTeX
@inproceedings{valencia2025aaai-ctd,
title = {{CTD4 - A Deep Continuous Distributional Actor-Critic Agent with a Kalman Fusion of Multiple Critics}},
author = {Valencia, David and Williams, Henry and Xing, Yuning and Gee, Trevor and MacDonald, Bruce A. and Liarokapis, Minas},
booktitle = {AAAI Conference on Artificial Intelligence},
year = {2025},
pages = {20956-20963},
doi = {10.1609/AAAI.V39I20.35391},
url = {https://mlanthology.org/aaai/2025/valencia2025aaai-ctd/}
}