Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910921456091136 |
|---|---|
| author | Salazar, Israfel Burda, Manuel Fernández Islam, Shayekh Bin Moakhar, Arshia Soltani Singh, Shivalika Farestam, Fabian Romanou, Angelika Boiko, Danylo Khullar, Dipika Zhang, Mike Krzemiński, Dominik Novikova, Jekaterina Shimabucoro, Luísa Imperial, Joseph Marvin Maheshwary, Rishabh Duwal, Sharad Amayuelas, Alfonso Rajwal, Swati Purbey, Jebish Ruby, Ahmed Popovič, Nicholas Suppa, Marek Wasi, Azmine Toushik Kadiyala, Ram Mohan Rao Tsymboi, Olga Kostritsya, Maksim Moakhar, Bardia Soltani Merlin, Gabriel da Costa Coletti, Otávio Ferracioli Shiviari, Maral Jabbari fard, MohammadAmin farahani Fernandez, Silvia Grandury, María Abulkhanov, Dmitry Sharma, Drishti De Mitri, Andre Guarnier Marchezi, Leticia Bossatto Heydari, Setayesh Obando-Ceron, Johan Kohut, Nazar Ermis, Beyza Elliott, Desmond Ferrante, Enzo Hooker, Sara Fadaee, Marzieh |
| author_facet | Salazar, Israfel Burda, Manuel Fernández Islam, Shayekh Bin Moakhar, Arshia Soltani Singh, Shivalika Farestam, Fabian Romanou, Angelika Boiko, Danylo Khullar, Dipika Zhang, Mike Krzemiński, Dominik Novikova, Jekaterina Shimabucoro, Luísa Imperial, Joseph Marvin Maheshwary, Rishabh Duwal, Sharad Amayuelas, Alfonso Rajwal, Swati Purbey, Jebish Ruby, Ahmed Popovič, Nicholas Suppa, Marek Wasi, Azmine Toushik Kadiyala, Ram Mohan Rao Tsymboi, Olga Kostritsya, Maksim Moakhar, Bardia Soltani Merlin, Gabriel da Costa Coletti, Otávio Ferracioli Shiviari, Maral Jabbari fard, MohammadAmin farahani Fernandez, Silvia Grandury, María Abulkhanov, Dmitry Sharma, Drishti De Mitri, Andre Guarnier Marchezi, Leticia Bossatto Heydari, Setayesh Obando-Ceron, Johan Kohut, Nazar Ermis, Beyza Elliott, Desmond Ferrante, Enzo Hooker, Sara Fadaee, Marzieh |
| contents | The evaluation of vision-language models (VLMs) has mainly relied on English-language benchmarks, leaving significant gaps in both multilingual and multicultural coverage. While multilingual benchmarks have expanded, both in size and languages, many rely on translations of English datasets, failing to capture cultural nuances. In this work, we propose Kaleidoscope, as the most comprehensive exam benchmark to date for the multilingual evaluation of vision-language models. Kaleidoscope is a large-scale, in-language multimodal benchmark designed to evaluate VLMs across diverse languages and visual inputs. Kaleidoscope covers 18 languages and 14 different subjects, amounting to a total of 20,911 multiple-choice questions. Built through an open science collaboration with a diverse group of researchers worldwide, Kaleidoscope ensures linguistic and cultural authenticity. We evaluate top-performing multilingual vision-language models and find that they perform poorly on low-resource languages and in complex multimodal scenarios. Our results highlight the need for progress on culturally inclusive multimodal evaluation frameworks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_07072 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation Salazar, Israfel Burda, Manuel Fernández Islam, Shayekh Bin Moakhar, Arshia Soltani Singh, Shivalika Farestam, Fabian Romanou, Angelika Boiko, Danylo Khullar, Dipika Zhang, Mike Krzemiński, Dominik Novikova, Jekaterina Shimabucoro, Luísa Imperial, Joseph Marvin Maheshwary, Rishabh Duwal, Sharad Amayuelas, Alfonso Rajwal, Swati Purbey, Jebish Ruby, Ahmed Popovič, Nicholas Suppa, Marek Wasi, Azmine Toushik Kadiyala, Ram Mohan Rao Tsymboi, Olga Kostritsya, Maksim Moakhar, Bardia Soltani Merlin, Gabriel da Costa Coletti, Otávio Ferracioli Shiviari, Maral Jabbari fard, MohammadAmin farahani Fernandez, Silvia Grandury, María Abulkhanov, Dmitry Sharma, Drishti De Mitri, Andre Guarnier Marchezi, Leticia Bossatto Heydari, Setayesh Obando-Ceron, Johan Kohut, Nazar Ermis, Beyza Elliott, Desmond Ferrante, Enzo Hooker, Sara Fadaee, Marzieh Computation and Language Computer Vision and Pattern Recognition The evaluation of vision-language models (VLMs) has mainly relied on English-language benchmarks, leaving significant gaps in both multilingual and multicultural coverage. While multilingual benchmarks have expanded, both in size and languages, many rely on translations of English datasets, failing to capture cultural nuances. In this work, we propose Kaleidoscope, as the most comprehensive exam benchmark to date for the multilingual evaluation of vision-language models. Kaleidoscope is a large-scale, in-language multimodal benchmark designed to evaluate VLMs across diverse languages and visual inputs. Kaleidoscope covers 18 languages and 14 different subjects, amounting to a total of 20,911 multiple-choice questions. Built through an open science collaboration with a diverse group of researchers worldwide, Kaleidoscope ensures linguistic and cultural authenticity. We evaluate top-performing multilingual vision-language models and find that they perform poorly on low-resource languages and in complex multimodal scenarios. Our results highlight the need for progress on culturally inclusive multimodal evaluation frameworks. |
| title | Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation |
| topic | Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2504.07072 |