Learning to Scale Multilingual Representations for Vision-Language Tasks
Abstract
Current multilingual vision-language models either require a large number of additional parameters for each supported language, or suffer performance degradation as languages are added. In this paper, we propose a Scalable Multilingual Aligned Language Representation (SMALR) that supports many languages with few model parameters without sacrificing downstream task performance. SMALR learns a fixed size language-agnostic representation for most words in a multilingual vocabulary, keeping language-specific features for just a few. We use a masked cross-language modeling loss to align features with context from other languages. Additionally, we propose a cross-lingual consistency module that ensures predictions made for a query and its machine translation are comparable. The effectiveness of SMALR is demonstrated with ten diverse languages, over twice the number supported in vision-language tasks to date. We evaluate on multilingual image-sentence retrieval and outperform prior work by 3-4% with less than 1/5th the training parameters compared to other word embedding methods.
Cite
Text
Burns et al. "Learning to Scale Multilingual Representations for Vision-Language Tasks." Proceedings of the European Conference on Computer Vision (ECCV), 2020. doi:10.1007/978-3-030-58548-8_12Markdown
[Burns et al. "Learning to Scale Multilingual Representations for Vision-Language Tasks." Proceedings of the European Conference on Computer Vision (ECCV), 2020.](https://mlanthology.org/eccv/2020/burns2020eccv-learning/) doi:10.1007/978-3-030-58548-8_12BibTeX
@inproceedings{burns2020eccv-learning,
title = {{Learning to Scale Multilingual Representations for Vision-Language Tasks}},
author = {Burns, Andrea and Kim, Donghyun and Wijaya, Derry and Saenko, Kate and Plummer, Bryan A.},
booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
year = {2020},
doi = {10.1007/978-3-030-58548-8_12},
url = {https://mlanthology.org/eccv/2020/burns2020eccv-learning/}
}