ST-PPO: A Spatio-Temporal Attention Enhanced Proximal Policy Optimization Algorithm for Autonomous Driving in Complex Traffic Scenarios

Abstract

Autonomous driving in complex traffic environments poses significant challenges due to the dynamic nature of multi-agent interactions and varying road conditions. This paper proposes ST-PPO, a novel reinforcement learning framework that integrates spatio-temporal attention mechanisms with Proximal Policy Optimization for autonomous vehicle control. The framework addresses three critical challenges: spatial feature extraction from complex traffic scenes, temporal dependency modeling of vehicle behaviors, and adaptive policy learning in dynamic environments. The spatial attention module captures crucial spatial relationships between traffic participants, while the temporal attention module models the sequential dependencies of driving behaviors. We evaluate our approach on challenging scenarios including merging areas, continuous curves, intersections, and adverse weather conditions. Comprehensive experiments demonstrate that ST-PPO significantly outperforms baseline methods, achieving 25.2% improvement in training efficiency and 7.6% increase in overall performance compared to vanilla PPO. The method demonstrates remarkable stability with 18.7% lower KL divergence and higher value estimation accuracy (0.8 explained variance). Ablation studies further validate the effectiveness of both spatial and temporal attention components. Our method shows particular strength in handling complex scenarios such as dense traffic and adverse weather conditions, where it maintains stable performance while baseline methods deteriorate significantly. The proposed approach represents a significant step toward robust autonomous driving systems capable of handling real-world traffic complexity.