DSTCFuse: A Method Based on Dual-Cycled Cross-Awareness of Structure Tensor for Semantic Segmentation via Infrared and Visible Image Fusion

Abstract

Multi-modality information fusion can compensate deficiencies of single modality and provide rich scene information for 2D semantic segmentation. However, the inconsistency in the feature space between different modalities may lead to poor presentation of objects and that would affect subsequent segmented effectiveness. The idea of modal transition can reduce the modal differences and avoid biased processing during the fusion process, but it is hard to perfectly retain the contents of the source images. To address these challenges, a fusion method based on dual-cycled cross-awareness of structure tensor is proposed. Firstly, we propose a dual-cycle modality transition network based on cross-awareness consistency to learn the differences in feature space from different modalities. Secondly, a set of globally structure-tensor preserving modules are designed to enhance the capabilities of the network in capturing complementary features and perceiving global modal consistency. Under the joint constraint of globally structure-tensor awareness loss and cross-awareness loss, our network achieves a robust mapping of feature space from visible to pseudo-infrared images without relying on Ground-Truth. Finally, the pseudo-infrared images that inherit the superior qualities of two modalities are fused with the original infrared images directly, which effectively reduces the complexity of fusion. Extensive comparative experiments show that our method outperforms state-of-the-art methods in qualitative and quantitative evaluation.