Palm: A Culturally Inclusive and Linguistically Diverse Dataset for Arabic LLMs
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918103751852032 |
|---|---|
| author | Alwajih, Fakhraddin Mekki, Abdellah El Magdy, Samar Mohamed Elmadany, Abdelrahim A. Nacar, Omer Nagoudi, El Moatez Billah Abdel-Salam, Reem Atwany, Hanin Nafea, Youssef Yahya, Abdulfattah Mohammed Alhamouri, Rahaf Alsayadi, Hamzah A. Zayed, Hiba Shatnawi, Sara Sibaee, Serry Ech-Chammakhy, Yasir Al-Dhabyani, Walid Ali, Marwa Mohamed Jarraya, Imen El-Shangiti, Ahmed Oumar Alraeesi, Aisha Al-Ghrawi, Mohammed Anwar Al-Batati, Abdulrahman S. Mohamed, Elgizouli Elgindi, Noha Taha Saeed, Muhammed Atou, Houdaifa Yahia, Issam Ait Bouayad, Abdelhak Machrouh, Mohammed Makouar, Amal Alkawi, Dania Mohamed, Mukhtar Abdelfadil, Safaa Taher Ounnoughene, Amine Ziad Anfel, Rouabhia Assi, Rwaa Sorkatti, Ahmed Tourad, Mohamedou Cheikh Koubaa, Anis Berrada, Ismail Jarrar, Mustafa Shehata, Shady Abdul-Mageed, Muhammad |
| author_facet | Alwajih, Fakhraddin Mekki, Abdellah El Magdy, Samar Mohamed Elmadany, Abdelrahim A. Nacar, Omer Nagoudi, El Moatez Billah Abdel-Salam, Reem Atwany, Hanin Nafea, Youssef Yahya, Abdulfattah Mohammed Alhamouri, Rahaf Alsayadi, Hamzah A. Zayed, Hiba Shatnawi, Sara Sibaee, Serry Ech-Chammakhy, Yasir Al-Dhabyani, Walid Ali, Marwa Mohamed Jarraya, Imen El-Shangiti, Ahmed Oumar Alraeesi, Aisha Al-Ghrawi, Mohammed Anwar Al-Batati, Abdulrahman S. Mohamed, Elgizouli Elgindi, Noha Taha Saeed, Muhammed Atou, Houdaifa Yahia, Issam Ait Bouayad, Abdelhak Machrouh, Mohammed Makouar, Amal Alkawi, Dania Mohamed, Mukhtar Abdelfadil, Safaa Taher Ounnoughene, Amine Ziad Anfel, Rouabhia Assi, Rwaa Sorkatti, Ahmed Tourad, Mohamedou Cheikh Koubaa, Anis Berrada, Ismail Jarrar, Mustafa Shehata, Shady Abdul-Mageed, Muhammad |
| contents | As large language models (LLMs) become increasingly integrated into daily life, ensuring their cultural sensitivity and inclusivity is paramount. We introduce our dataset, a year-long community-driven project covering all 22 Arab countries. The dataset includes instructions (input, response pairs) in both Modern Standard Arabic (MSA) and dialectal Arabic (DA), spanning 20 diverse topics. Built by a team of 44 researchers across the Arab world, all of whom are authors of this paper, our dataset offers a broad, inclusive perspective. We use our dataset to evaluate the cultural and dialectal capabilities of several frontier LLMs, revealing notable limitations. For instance, while closed-source LLMs generally exhibit strong performance, they are not without flaws, and smaller open-source models face greater challenges. Moreover, certain countries (e.g., Egypt, the UAE) appear better represented than others (e.g., Iraq, Mauritania, Yemen). Our annotation guidelines, code, and data for reproducibility are publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_00151 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Palm: A Culturally Inclusive and Linguistically Diverse Dataset for Arabic LLMs Alwajih, Fakhraddin Mekki, Abdellah El Magdy, Samar Mohamed Elmadany, Abdelrahim A. Nacar, Omer Nagoudi, El Moatez Billah Abdel-Salam, Reem Atwany, Hanin Nafea, Youssef Yahya, Abdulfattah Mohammed Alhamouri, Rahaf Alsayadi, Hamzah A. Zayed, Hiba Shatnawi, Sara Sibaee, Serry Ech-Chammakhy, Yasir Al-Dhabyani, Walid Ali, Marwa Mohamed Jarraya, Imen El-Shangiti, Ahmed Oumar Alraeesi, Aisha Al-Ghrawi, Mohammed Anwar Al-Batati, Abdulrahman S. Mohamed, Elgizouli Elgindi, Noha Taha Saeed, Muhammed Atou, Houdaifa Yahia, Issam Ait Bouayad, Abdelhak Machrouh, Mohammed Makouar, Amal Alkawi, Dania Mohamed, Mukhtar Abdelfadil, Safaa Taher Ounnoughene, Amine Ziad Anfel, Rouabhia Assi, Rwaa Sorkatti, Ahmed Tourad, Mohamedou Cheikh Koubaa, Anis Berrada, Ismail Jarrar, Mustafa Shehata, Shady Abdul-Mageed, Muhammad Computation and Language Artificial Intelligence As large language models (LLMs) become increasingly integrated into daily life, ensuring their cultural sensitivity and inclusivity is paramount. We introduce our dataset, a year-long community-driven project covering all 22 Arab countries. The dataset includes instructions (input, response pairs) in both Modern Standard Arabic (MSA) and dialectal Arabic (DA), spanning 20 diverse topics. Built by a team of 44 researchers across the Arab world, all of whom are authors of this paper, our dataset offers a broad, inclusive perspective. We use our dataset to evaluate the cultural and dialectal capabilities of several frontier LLMs, revealing notable limitations. For instance, while closed-source LLMs generally exhibit strong performance, they are not without flaws, and smaller open-source models face greater challenges. Moreover, certain countries (e.g., Egypt, the UAE) appear better represented than others (e.g., Iraq, Mauritania, Yemen). Our annotation guidelines, code, and data for reproducibility are publicly available. |
| title | Palm: A Culturally Inclusive and Linguistically Diverse Dataset for Arabic LLMs |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2503.00151 |