Visual Commonsense Representation Learning via Causal Inference
Abstract
We present a novel unsupervised feature representation learning method, Visual Commonsense Region-based Convolutional Neural Network (VC R-CNN1), to serve as an improved visual region encoder for high-level tasks such as captioning and VQA. Given a set of detected object regions in an image (e.g., using Faster R-CNN), like any other unsupervised feature learning methods (e.g., word2vec), the proxy training objective of VC R-CNN is to predict the contextual objects of a region. However, they are fundamentally different: the prediction of VC R-CNN is by using causal intervention: P(Y|do(X)), while others are by using the conventional likelihood: P(Y|X). We extensively apply VC R-CNN features in prevailing models of two popular tasks: Image Captioning and VQA, and observe consistent performance boosts across all the methods, achieving many new state-of-the-arts2.
Cite
Text
Wang et al. "Visual Commonsense Representation Learning via Causal Inference." IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020. doi:10.1109/CVPRW50498.2020.00197Markdown
[Wang et al. "Visual Commonsense Representation Learning via Causal Inference." IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020.](https://mlanthology.org/cvprw/2020/wang2020cvprw-visual/) doi:10.1109/CVPRW50498.2020.00197BibTeX
@inproceedings{wang2020cvprw-visual,
title = {{Visual Commonsense Representation Learning via Causal Inference}},
author = {Wang, Tan and Huang, Jianqiang and Zhang, Hanwang and Sun, Qianru},
booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops},
year = {2020},
pages = {1547-1550},
doi = {10.1109/CVPRW50498.2020.00197},
url = {https://mlanthology.org/cvprw/2020/wang2020cvprw-visual/}
}