Connecting Language to Images: A Progressive Attention-Guided Network for Simultaneous Image Captioning and Language Grounding
Abstract
Image captioning and visual language grounding are two important tasks for image understanding, but are seldom considered together. In this paper, we propose a Progressive Attention-Guided Network (PAGNet), which simultaneously generates image captions and predicts bounding boxes for caption words. PAGNet mainly has two distinctive properties: i) It can progressively refine the predictive results of image captioning, by updating the attention map with the predicted bounding boxes. ii) It learns bounding boxes of the words using a weakly supervised strategy, which combines the frameworks of Multiple Instance Learning (MIL) and Markov Decision Process (MDP). By using the attention map generated in the captioning process, PAGNet significantly reduces the search space of the MDP. We conduct experiments on benchmark datasets to demonstrate the effectiveness of PAGNet and results show that PAGNet achieves the best performance.
Cite
Text
Song et al. "Connecting Language to Images: A Progressive Attention-Guided Network for Simultaneous Image Captioning and Language Grounding." AAAI Conference on Artificial Intelligence, 2019. doi:10.1609/AAAI.V33I01.33018885Markdown
[Song et al. "Connecting Language to Images: A Progressive Attention-Guided Network for Simultaneous Image Captioning and Language Grounding." AAAI Conference on Artificial Intelligence, 2019.](https://mlanthology.org/aaai/2019/song2019aaai-connecting/) doi:10.1609/AAAI.V33I01.33018885BibTeX
@inproceedings{song2019aaai-connecting,
title = {{Connecting Language to Images: A Progressive Attention-Guided Network for Simultaneous Image Captioning and Language Grounding}},
author = {Song, Lingyun and Liu, Jun and Qian, Buyue and Chen, Yihe},
booktitle = {AAAI Conference on Artificial Intelligence},
year = {2019},
pages = {8885-8892},
doi = {10.1609/AAAI.V33I01.33018885},
url = {https://mlanthology.org/aaai/2019/song2019aaai-connecting/}
}