Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.10497 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910968391401472 |
|---|---|
| author | Xuan, Weihao Yang, Rui Qi, Heli Zeng, Qingcheng Xiao, Yunze Feng, Aosong Liu, Dairui Xing, Yun Wang, Junjue Gao, Fan Lu, Jinghui Jiang, Yuang Li, Huitao Li, Xin Yu, Kunyu Dong, Ruihai Gu, Shangding Li, Yuekang Xie, Xiaofei Juefei-Xu, Felix Khomh, Foutse Yoshie, Osamu Chen, Qingyu Teodoro, Douglas Liu, Nan Goebel, Randy Ma, Lei Marrese-Taylor, Edison Lu, Shijian Iwasawa, Yusuke Matsuo, Yutaka Li, Irene |
| author_facet | Xuan, Weihao Yang, Rui Qi, Heli Zeng, Qingcheng Xiao, Yunze Feng, Aosong Liu, Dairui Xing, Yun Wang, Junjue Gao, Fan Lu, Jinghui Jiang, Yuang Li, Huitao Li, Xin Yu, Kunyu Dong, Ruihai Gu, Shangding Li, Yuekang Xie, Xiaofei Juefei-Xu, Felix Khomh, Foutse Yoshie, Osamu Chen, Qingyu Teodoro, Douglas Liu, Nan Goebel, Randy Ma, Lei Marrese-Taylor, Edison Lu, Shijian Iwasawa, Yusuke Matsuo, Yutaka Li, Irene |
| contents | Existing large language model (LLM) evaluation benchmarks primarily focus on English, while current multilingual tasks lack parallel questions that specifically assess cross-linguistic reasoning abilities. This dual limitation makes it challenging to comprehensively assess LLMs' performance in the multilingual setting. To fill this gap, we introduce MMLU-ProX, a comprehensive benchmark covering 29 languages, built on an English benchmark. Each language version consists of 11,829 identical questions, enabling direct cross-linguistic comparisons. Additionally, to meet efficient evaluation needs, we provide a lite version containing 658 questions per language. To ensure the high quality of MMLU-ProX, we employ a rigorous development process that involves multiple powerful LLMs for translation, followed by expert review to ensure accurate expression, consistent terminology, and cultural relevance. Building on this, we systematically evaluate 36 state-of-the-art LLMs, including reasoning-enhanced and multilingual-optimized LLMs. The results reveal significant disparities in the multilingual capabilities of LLMs: While they perform well in high-resource languages, their performance declines markedly in low-resource languages, with gaps of up to 24.3%. Through MMLU-ProX, we aim to advance the development of more inclusive AI systems and promote equitable access to technology across global contexts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_10497 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation Xuan, Weihao Yang, Rui Qi, Heli Zeng, Qingcheng Xiao, Yunze Feng, Aosong Liu, Dairui Xing, Yun Wang, Junjue Gao, Fan Lu, Jinghui Jiang, Yuang Li, Huitao Li, Xin Yu, Kunyu Dong, Ruihai Gu, Shangding Li, Yuekang Xie, Xiaofei Juefei-Xu, Felix Khomh, Foutse Yoshie, Osamu Chen, Qingyu Teodoro, Douglas Liu, Nan Goebel, Randy Ma, Lei Marrese-Taylor, Edison Lu, Shijian Iwasawa, Yusuke Matsuo, Yutaka Li, Irene Computation and Language Existing large language model (LLM) evaluation benchmarks primarily focus on English, while current multilingual tasks lack parallel questions that specifically assess cross-linguistic reasoning abilities. This dual limitation makes it challenging to comprehensively assess LLMs' performance in the multilingual setting. To fill this gap, we introduce MMLU-ProX, a comprehensive benchmark covering 29 languages, built on an English benchmark. Each language version consists of 11,829 identical questions, enabling direct cross-linguistic comparisons. Additionally, to meet efficient evaluation needs, we provide a lite version containing 658 questions per language. To ensure the high quality of MMLU-ProX, we employ a rigorous development process that involves multiple powerful LLMs for translation, followed by expert review to ensure accurate expression, consistent terminology, and cultural relevance. Building on this, we systematically evaluate 36 state-of-the-art LLMs, including reasoning-enhanced and multilingual-optimized LLMs. The results reveal significant disparities in the multilingual capabilities of LLMs: While they perform well in high-resource languages, their performance declines markedly in low-resource languages, with gaps of up to 24.3%. Through MMLU-ProX, we aim to advance the development of more inclusive AI systems and promote equitable access to technology across global contexts. |
| title | MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2503.10497 |