VeGAN: Using GANs for Augmentation in Latent Space to Improve the Semantic Segmentation of Vehicles in Images from an Aerial Perspective

Abstract

Generative Adversarial Networks (GANs) are a new network architecture capable of delivering state-of-the-art performance in generating synthetic images in various domains. We train a network called VeGAN (Vehicle Generative Adversarial Network) to generate realistic images of vehicles that look like images taken from a top-down view of an unmanned aerial vehicle (UAV). The generated images are used as additional training data for a semantic segmentation network, which precisely detects vehicles in recordings of traffic on highways. While images are commonly generated randomly for a content-based augmentation, we leverage ideas from the domain of active learning. Using a network which is based on the InfoGAN architecture allows mapping existing vehicle images to a latent space representation. After mapping the complete training dataset, we perform the augmentation in the latent space. The applied techniques include creating variations of given hard negative samples and generating samples in sparsely occupied areas of the latent space. We improve the IoU of the semantic segmentation network from 93.4% to 94.9% and reduce the mean positional error of the detected vehicles' centers from 0.51 to 0.37 pixels in longitudinal and from 0.21 to 0.17 pixels in lateral direction.