Learning with Side Information Through Modality Hallucination

Judy Hoffman, Saurabh Gupta, Trevor Darrell

CVPR 2016

doi:10.1109/CVPR.2016.96 /cvpr/2016/hoffman2016cvpr-learning/

Abstract

We present a modality hallucination architecture for training an RGB object detection model which incorporates depth side information at training time. Our convolutional hallucination network learns a new and complementary RGB image representation which is taught to mimic convolutional mid-level features from a depth network. At test time images are processed jointly through the RGB and hallucination networks to produce improved detection performance. Thus, our method transfers information commonly extracted from depth training data to a network which can extract that information from the RGB counterpart. We present results on the standard NYUDv2 dataset and report improvement on the RGB detection task.

PDF CVPR Semantic Scholar

Cite

Text

Hoffman et al. "Learning with Side Information Through Modality Hallucination." Conference on Computer Vision and Pattern Recognition, 2016. doi:10.1109/CVPR.2016.96

Markdown

[Hoffman et al. "Learning with Side Information Through Modality Hallucination." Conference on Computer Vision and Pattern Recognition, 2016.](https://mlanthology.org/cvpr/2016/hoffman2016cvpr-learning/) doi:10.1109/CVPR.2016.96

BibTeX

@inproceedings{hoffman2016cvpr-learning,
  title     = {{Learning with Side Information Through Modality Hallucination}},
  author    = {Hoffman, Judy and Gupta, Saurabh and Darrell, Trevor},
  booktitle = {Conference on Computer Vision and Pattern Recognition},
  year      = {2016},
  doi       = {10.1109/CVPR.2016.96},
  url       = {https://mlanthology.org/cvpr/2016/hoffman2016cvpr-learning/}
}