Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.21910 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909555831603200 |
|---|---|
| author | Kadaoui, Karima Atwany, Hanin Al-Ali, Hamdan Mohamed, Abdelrahman Mekky, Ali Tilga, Sergei Fedorova, Natalia Artemova, Ekaterina Aldarmaki, Hanan Kementchedjhieva, Yova |
| author_facet | Kadaoui, Karima Atwany, Hanin Al-Ali, Hamdan Mohamed, Abdelrahman Mekky, Ali Tilga, Sergei Fedorova, Natalia Artemova, Ekaterina Aldarmaki, Hanan Kementchedjhieva, Yova |
| contents | We introduce JEEM, a benchmark designed to evaluate Vision-Language Models (VLMs) on visual understanding across four Arabic-speaking countries: Jordan, The Emirates, Egypt, and Morocco. JEEM includes the tasks of image captioning and visual question answering, and features culturally rich and regionally diverse content. This dataset aims to assess the ability of VLMs to generalize across dialects and accurately interpret cultural elements in visual contexts. In an evaluation of five prominent open-source Arabic VLMs and GPT-4V, we find that the Arabic VLMs consistently underperform, struggling with both visual understanding and dialect-specific generation. While GPT-4V ranks best in this comparison, the model's linguistic competence varies across dialects, and its visual understanding capabilities lag behind. This underscores the need for more inclusive models and the value of culturally-diverse evaluation paradigms. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_21910 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | JEEM: Vision-Language Understanding in Four Arabic Dialects Kadaoui, Karima Atwany, Hanin Al-Ali, Hamdan Mohamed, Abdelrahman Mekky, Ali Tilga, Sergei Fedorova, Natalia Artemova, Ekaterina Aldarmaki, Hanan Kementchedjhieva, Yova Computation and Language Artificial Intelligence We introduce JEEM, a benchmark designed to evaluate Vision-Language Models (VLMs) on visual understanding across four Arabic-speaking countries: Jordan, The Emirates, Egypt, and Morocco. JEEM includes the tasks of image captioning and visual question answering, and features culturally rich and regionally diverse content. This dataset aims to assess the ability of VLMs to generalize across dialects and accurately interpret cultural elements in visual contexts. In an evaluation of five prominent open-source Arabic VLMs and GPT-4V, we find that the Arabic VLMs consistently underperform, struggling with both visual understanding and dialect-specific generation. While GPT-4V ranks best in this comparison, the model's linguistic competence varies across dialects, and its visual understanding capabilities lag behind. This underscores the need for more inclusive models and the value of culturally-diverse evaluation paradigms. |
| title | JEEM: Vision-Language Understanding in Four Arabic Dialects |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2503.21910 |