M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-Training

Minheng Ni, Haoyang Huang, Lin Su, Edward Cui, Taroon Bharti, Lijuan Wang, Dongdong Zhang, Nan Duan

CVPR 2021 pp. 3977-3986

doi:10.1109/CVPR46437.2021.00397 /cvpr/2021/ni2021cvpr-m3p/

Abstract

We present M3P, a Multitask Multilingual Multimodal Pre-trained model that combines multilingual pre-training and multimodal pre-training into a unified framework via multitask pre-training. Our goal is to learn universal representations that can map objects occurred in different modalities or texts expressed in different languages into a common semantic space. In addition, to explicitly encourage fine-grained alignment between images and non-English languages, we also propose Multimodal Code-switched Training (MCT) to combine monolingual pre-training and multimodal pre-training via a code-switch strategy. Experiments are performed on the multilingual image retrieval task across two benchmark datasets, including MSCOCO and Multi30K. M3P can achieve comparable results for English and new state-of-the-art results for non-English languages.

PDF CVPR Semantic Scholar

Cite

Text

Ni et al. "M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-Training." Conference on Computer Vision and Pattern Recognition, 2021. doi:10.1109/CVPR46437.2021.00397

Markdown

[Ni et al. "M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-Training." Conference on Computer Vision and Pattern Recognition, 2021.](https://mlanthology.org/cvpr/2021/ni2021cvpr-m3p/) doi:10.1109/CVPR46437.2021.00397

BibTeX

@inproceedings{ni2021cvpr-m3p,
  title     = {{M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-Training}},
  author    = {Ni, Minheng and Huang, Haoyang and Su, Lin and Cui, Edward and Bharti, Taroon and Wang, Lijuan and Zhang, Dongdong and Duan, Nan},
  booktitle = {Conference on Computer Vision and Pattern Recognition},
  year      = {2021},
  pages     = {3977-3986},
  doi       = {10.1109/CVPR46437.2021.00397},
  url       = {https://mlanthology.org/cvpr/2021/ni2021cvpr-m3p/}
}