Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.12788 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915399564525568 |
|---|---|
| author | Sadallah, Abdelrahman Tonga, Junior Cedric Almubarak, Khalid Almheiri, Saeed Atif, Farah Qwaider, Chatrine Kadaoui, Karima Shatnawi, Sara Alesh, Yaser Koto, Fajri |
| author_facet | Sadallah, Abdelrahman Tonga, Junior Cedric Almubarak, Khalid Almheiri, Saeed Atif, Farah Qwaider, Chatrine Kadaoui, Karima Shatnawi, Sara Alesh, Yaser Koto, Fajri |
| contents | Despite progress in Arabic large language models, such as Jais and AceGPT, their evaluation on commonsense reasoning has largely relied on machine-translated datasets, which lack cultural depth and may introduce Anglocentric biases. Commonsense reasoning is shaped by geographical and cultural contexts, and existing English datasets fail to capture the diversity of the Arab world. To address this, we introduce ArabCulture, a commonsense reasoning dataset in Modern Standard Arabic (MSA), covering cultures of 13 countries across the Gulf, Levant, North Africa, and the Nile Valley. The dataset was built from scratch by engaging native speakers to write and validate culturally relevant questions for their respective countries. ArabCulture spans 12 daily life domains with 54 fine-grained subtopics, reflecting various aspects of social norms, traditions, and everyday experiences. Zero-shot evaluations show that open-weight language models with up to 32B parameters struggle to comprehend diverse Arab cultures, with performance varying across regions. These findings highlight the need for more culturally aware models and datasets tailored to the Arabic-speaking world. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_12788 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Commonsense Reasoning in Arab Culture Sadallah, Abdelrahman Tonga, Junior Cedric Almubarak, Khalid Almheiri, Saeed Atif, Farah Qwaider, Chatrine Kadaoui, Karima Shatnawi, Sara Alesh, Yaser Koto, Fajri Computation and Language Despite progress in Arabic large language models, such as Jais and AceGPT, their evaluation on commonsense reasoning has largely relied on machine-translated datasets, which lack cultural depth and may introduce Anglocentric biases. Commonsense reasoning is shaped by geographical and cultural contexts, and existing English datasets fail to capture the diversity of the Arab world. To address this, we introduce ArabCulture, a commonsense reasoning dataset in Modern Standard Arabic (MSA), covering cultures of 13 countries across the Gulf, Levant, North Africa, and the Nile Valley. The dataset was built from scratch by engaging native speakers to write and validate culturally relevant questions for their respective countries. ArabCulture spans 12 daily life domains with 54 fine-grained subtopics, reflecting various aspects of social norms, traditions, and everyday experiences. Zero-shot evaluations show that open-weight language models with up to 32B parameters struggle to comprehend diverse Arab cultures, with performance varying across regions. These findings highlight the need for more culturally aware models and datasets tailored to the Arabic-speaking world. |
| title | Commonsense Reasoning in Arab Culture |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.12788 |