Foundation Model-Based Data Selection for Dense Prediction Tasks

Abstract

Data selection, the problem of selecting a small dataset to be labeled from a large unlabeled pool is an important practical problem. In particular, dense prediction tasks such as object detection and segmentation require high-quality labels at pixel level, which are particularly costly to obtain. We propose object-focused data selection (OFDS) which leverages object-level representations from foundation models to ensure that the selected image subsets semantically cover the target classes, including rare ones. We show that OFDS achieves state-of-the-art performance both for object detection and image segmentation with substantial improvements over all baselines in scenarios with imbalanced class distributions. Moreover, we demonstrate that pre-training with autolabels from foundation models on the full datasets before fine-tuning on human-labeled subsets selected by OFDS further enhances the final performance. Finally, OFDS consistently improves active learning methods when replacing the random selection of the initial labeled dataset, the so-called "cold start problem'' of active learning, with OFDS.