iIPPC-V2X: Multi-Modality Fusion Perception System for Cooperative Vehicle Infrastructure System with Self-Supervised Learning

Abstract

In the cooperative vehicle infrastructure system (CVIS), perception data comes from multiple sources and modalities. Designing an optimal fusion mechanism to achieve more accurate perception is a key area of focus. In this paper, a multiple sources and multiple modalities self-supervised learning fusion perception framework (iIPPC-V2X) based on Invariant Linear Probabilistic Population Code(iIPPC) mechanism is proposed for vehicle-to-road cooperation. By simulating the neural integration mechanism of living organisms, the framework adaptively adjusts the weights of sensory information to achieve near-Bayesian optimal information integration. This paper also designs a vehicle infrastructure multiple sources and multiple modalities information fusion adjustment module to deal with the influence of external interference on the perception system, and evaluates the perception effect of the system based on mutual information theory. In addition, this paper proposes a context-aware self-supervised learning method based on large language model (LLM), which improves the robustness and adaptability of the perception model by augmenting data. The experimental results show that the proposed method significantly improves the accuracy and robustness of the cooperative vehicle infrastructure perception system in complex perception environments.