Kaleidoscope: In-Language Exams for Massively Multilingual Vision Evaluation
Abstract
The evaluation of vision-language models (VLMs) has mainly relied on English-language benchmarks, leaving significant gaps in both multilingual and multicultural coverage. While multilingual benchmarks have expanded, both in size and language, many rely on translations of English datasets, failing to capture cultural nuances. In this work, we propose Kaleidoscope, as the most comprehensive exam benchmark to date for the multilingual evaluation of vision-language models. Kaleidoscope is a large-scale, in-language multimodal benchmark designed to evaluate VLMs across diverse languages and visual inputs. Kaleidoscope covers 18 languages and 14 different subjects, amounting to a total of 20,911 multiple-choice questions. Built through an open science collaboration with a diverse group of researchers worldwide, Kaleidoscope ensures linguistic and cultural authenticity. We evaluate top-performing multilingual vision-language models and find that they perform poorly on low-resource languages and in complex multimodal scenarios. Our results highlight the need for progress on culturally inclusive multimodal evaluation frameworks.
Cite
Text
Salazar et al. "Kaleidoscope: In-Language Exams for Massively Multilingual Vision Evaluation." International Conference on Learning Representations, 2026.Markdown
[Salazar et al. "Kaleidoscope: In-Language Exams for Massively Multilingual Vision Evaluation." International Conference on Learning Representations, 2026.](https://mlanthology.org/iclr/2026/salazar2026iclr-kaleidoscope/)BibTeX
@inproceedings{salazar2026iclr-kaleidoscope,
title = {{Kaleidoscope: In-Language Exams for Massively Multilingual Vision Evaluation}},
author = {Salazar, Israfel and Burda, Manuel Fernández and Bin Islam, Shayekh and Moakhar, Arshia Soltani and Singh, Shivalika and Farestam, Fabian and Romanou, Angelika and Boiko, Danylo and Khullar, Dipika and Zhang, Mike and Krzemiński, Dominik and Novikova, Jekaterina and Shimabucoro, Luísa and Imperial, Joseph Marvin and Maheshwary, Rishabh and Duwal, Sharad and Amayuelas, Alfonso and Rajwal, Swati and Purbey, Jebish and Ruby, Ahmed and Popovič, Nicholas and Suppa, Marek and Wasi, Azmine Toushik and Kadiyala, Ram Mohan Rao and Tsymboi, Olga and Kostritsya, Maksim and Moakhar, Bardia soltani and da Costa Merlin, Gabriel and Coletti, Otávio Ferracioli and Jabbarishiviari, Maral and Farahanifard, Mohammadamin and Fernandez, Silvia Andrea and Grandury, María and Abulkhanov, Dmitry and Sharma, Drishti and De Mitri, Andre Guarnier and Marchezi, Leticia Bossatto and Heydari, Setayesh and Obando-Ceron, Johan and Kohut, Nazar and Ermis, Beyza and Elliott, Desmond and Ferrante, Enzo and Hooker, Sara and Fadaee, Marzieh},
booktitle = {International Conference on Learning Representations},
year = {2026},
url = {https://mlanthology.org/iclr/2026/salazar2026iclr-kaleidoscope/}
}