Sensor Equivariance: A Framework for Semantic Segmentation with Diverse Camera Models

Abstract

Objects are represented differently in projection-based sensors such as cameras depending on sensor resolution, field of view, and distortion, leading to distorted physical and geometric properties. As a result, sensor data processing depend on these properties. With the large variations of sensors on the market, an equivariant representation and suitable processing are necessary to become independent of the sensor used. In this work, we propose an extension of conventional image data by an additional channel in which the associated projection properties are encoded. Furthermore, we introduce a SensorConv layer as an extension to the conventional convolution layer. SensorConv enable using projection properties in convolutional neural networks. To that end, we propose an architecture for using the SensorConv layer in the Detectron2 [21] framework. We collected a dataset of equirectangular images for our experiments with the CARLA [3] simulator. To analyze multiple sensor models (i.e., sensor intrinsic), we created an augmentation method to emulate a high variability of sensors from the collected equirectangular panoramas. In our experiment, we show that our method can generalize better across different camera sensors.