Learning Transferrable Knowledge for Semantic Segmentation with Deep Convolutional Neural Network

Abstract

We propose a novel weakly-supervised semantic segmentation algorithm based on Deep Convolutional Neural Net- work (DCNN). Contrary to existing weakly-supervised approaches, our algorithm exploits auxiliary segmentation an- notations available for different categories to guide segmentations on images with only image-level class labels. To make segmentation knowledge transferrable across categories, we design a decoupled encoder-decoder architecture with attention model. In this architecture, the model generates spatial highlights of each category presented in images using an attention model, and subsequently per- forms binary segmentation for each highlighted region using decoder. Combining attention model, the decoder trained with segmentation annotations in different categories boosts accuracy of weakly-supervised semantic segmentation. The proposed algorithm demonstrates substantially improved performance compared to the state-of-the- art weakly-supervised techniques in PASCAL VOC 2012 dataset when our model is trained with the annotations in 60 exclusive categories in Microsoft COCO dataset.