Are Arabic Benchmarks Reliable? QIMMA's Quality-First Approach to LLM Evaluation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917383726170112 |
|---|---|
| author | AlQadi, Leen Alzubaidi, Ahmed Alyafeai, Mohammed Alobeidli, Hamza Alhammadi, Maitha Alsuwaidi, Shaikha Alkaabi, Omar Boussaha, Basma El Amel Hacid, Hakim |
| author_facet | AlQadi, Leen Alzubaidi, Ahmed Alyafeai, Mohammed Alobeidli, Hamza Alhammadi, Maitha Alsuwaidi, Shaikha Alkaabi, Omar Boussaha, Basma El Amel Hacid, Hakim |
| contents | We present QIMMA, a quality-assured Arabic LLM leaderboard that places systematic benchmark validation at its core. Rather than aggregating existing resources as-is, QIMMA applies a multi-model assessment pipeline combining automated LLM judgment with human review to surface and resolve systematic quality issues in well-established Arabic benchmarks before evaluation. The result is a curated, multi-domain, multi-task evaluation suite of over 52k samples, grounded predominantly in native Arabic content; code evaluation tasks are the sole exception, as they are inherently language-agnostic. Transparent implementation via LightEval, EvalPlus and public release of per-sample inference outputs make QIMMA a reproducible and community-extensible foundation for Arabic NLP evaluation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_03395 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Are Arabic Benchmarks Reliable? QIMMA's Quality-First Approach to LLM Evaluation AlQadi, Leen Alzubaidi, Ahmed Alyafeai, Mohammed Alobeidli, Hamza Alhammadi, Maitha Alsuwaidi, Shaikha Alkaabi, Omar Boussaha, Basma El Amel Hacid, Hakim Computation and Language We present QIMMA, a quality-assured Arabic LLM leaderboard that places systematic benchmark validation at its core. Rather than aggregating existing resources as-is, QIMMA applies a multi-model assessment pipeline combining automated LLM judgment with human review to surface and resolve systematic quality issues in well-established Arabic benchmarks before evaluation. The result is a curated, multi-domain, multi-task evaluation suite of over 52k samples, grounded predominantly in native Arabic content; code evaluation tasks are the sole exception, as they are inherently language-agnostic. Transparent implementation via LightEval, EvalPlus and public release of per-sample inference outputs make QIMMA a reproducible and community-extensible foundation for Arabic NLP evaluation. |
| title | Are Arabic Benchmarks Reliable? QIMMA's Quality-First Approach to LLM Evaluation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2604.03395 |