Evaluating Arabic Large Language Models: A Survey of Benchmarks, Methods, and Gaps
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908596531363840 |
|---|---|
| author | Alzubaidi, Ahmed Alsuwaidi, Shaikha Boussaha, Basma El Amel AlQadi, Leen Alkaabi, Omar Alyafeai, Mohammed Alobeidli, Hamza Hacid, Hakim |
| author_facet | Alzubaidi, Ahmed Alsuwaidi, Shaikha Boussaha, Basma El Amel AlQadi, Leen Alkaabi, Omar Alyafeai, Mohammed Alobeidli, Hamza Hacid, Hakim |
| contents | This survey provides the first systematic review of Arabic LLM benchmarks, analyzing 40+ evaluation benchmarks across NLP tasks, knowledge domains, cultural understanding, and specialized capabilities. We propose a taxonomy organizing benchmarks into four categories: Knowledge, NLP Tasks, Culture and Dialects, and Target-Specific evaluations. Our analysis reveals significant progress in benchmark diversity while identifying critical gaps: limited temporal evaluation, insufficient multi-turn dialogue assessment, and cultural misalignment in translated datasets. We examine three primary approaches: native collection, translation, and synthetic generation discussing their trade-offs regarding authenticity, scale, and cost. This work serves as a comprehensive reference for Arabic NLP researchers, providing insights into benchmark methodologies, reproducibility standards, and evaluation metrics while offering recommendations for future development. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_13430 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Evaluating Arabic Large Language Models: A Survey of Benchmarks, Methods, and Gaps Alzubaidi, Ahmed Alsuwaidi, Shaikha Boussaha, Basma El Amel AlQadi, Leen Alkaabi, Omar Alyafeai, Mohammed Alobeidli, Hamza Hacid, Hakim Computation and Language This survey provides the first systematic review of Arabic LLM benchmarks, analyzing 40+ evaluation benchmarks across NLP tasks, knowledge domains, cultural understanding, and specialized capabilities. We propose a taxonomy organizing benchmarks into four categories: Knowledge, NLP Tasks, Culture and Dialects, and Target-Specific evaluations. Our analysis reveals significant progress in benchmark diversity while identifying critical gaps: limited temporal evaluation, insufficient multi-turn dialogue assessment, and cultural misalignment in translated datasets. We examine three primary approaches: native collection, translation, and synthetic generation discussing their trade-offs regarding authenticity, scale, and cost. This work serves as a comprehensive reference for Arabic NLP researchers, providing insights into benchmark methodologies, reproducibility standards, and evaluation metrics while offering recommendations for future development. |
| title | Evaluating Arabic Large Language Models: A Survey of Benchmarks, Methods, and Gaps |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2510.13430 |