Point-Supervised Panoptic Segmentation via Estimating Pseudo Labels from Learnable Distance

Abstract

To bridge the gap between point labels and per-pixel labels, existing point-supervised panoptic segmentation methods usually estimate dense pseudo labels by assigning unlabeled pixels to corresponding instances according to rule-based pixel-to-instance distances. These distances cannot be optimized by point labels end to end and are usually suboptimal, which result in inaccurate pseudo labels. Here we propose to assign unlabeled pixels to corresponding instances based on a learnable distance. Specifically, we represent each instance as an anchor query, then predict the pixel-to-instance distance based on the cross-attention between anchor queries and pixel features through a distance branch, the predicted distance is supervised by point labels end to end. In order that each query can accurately represent the corresponding instance, we iteratively improve anchor queries through query aggregating and query enhancing processes, then improved distance results and pseudo labels are predicted with these queries. We have experimentally demonstrated the effectiveness of our approach and achieved state-of-the-art results.