Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generation

Abstract

Generation of synthetic data has allowed Machine Learning practitioners to bypass the need for costly collection and labeling of large datasets. Unfortunately the generation of such data often requires experts to carefully design sampling procedures that guarantee creation of realistic scenes. These sampling procedures typically require experts to specify certain structural aspects such as scene layout information. This is often hard to design, as scenes are generally highly complex and diverse. In this paper, we propose a generative model of synthetic scenes that reduces the distribution gap between the scene structure of generated scenes and a real target image dataset. Importantly, since labeling scene structures for real images is incredibly cumbersome, our method operates without any ground truth structure information for real data. Experiments on two synthetic datasets and a real driving dataset show that the method successfully bridges the distribution gap of discrete structural features between real and generated images.