BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912336604823552 |
|---|---|
| author | Huang, Xu Zhu, Wenhao Hu, Hanxu He, Conghui Li, Lei Huang, Shujian Yuan, Fei |
| author_facet | Huang, Xu Zhu, Wenhao Hu, Hanxu He, Conghui Li, Lei Huang, Shujian Yuan, Fei |
| contents | Previous multilingual benchmarks focus primarily on simple understanding tasks, but for large language models(LLMs), we emphasize proficiency in instruction following, reasoning, long context understanding, code generation, and so on. However, measuring these advanced capabilities across languages is underexplored. To address the disparity, we introduce BenchMAX, a multi-way multilingual evaluation benchmark that allows for fair comparisons of these important abilities across languages. To maintain high quality, three distinct native-speaking annotators independently annotate each sample within all tasks after the data was machine-translated from English into 16 other languages. Additionally, we present a novel translation challenge stemming from dataset construction. Extensive experiments on BenchMAX reveal varying effectiveness of core capabilities across languages, highlighting performance gaps that cannot be bridged by simply scaling up model size. BenchMAX serves as a comprehensive multilingual evaluation platform, providing a promising test bed to promote the development of multilingual language models. The dataset and code are publicly accessible. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_07346 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models Huang, Xu Zhu, Wenhao Hu, Hanxu He, Conghui Li, Lei Huang, Shujian Yuan, Fei Computation and Language Previous multilingual benchmarks focus primarily on simple understanding tasks, but for large language models(LLMs), we emphasize proficiency in instruction following, reasoning, long context understanding, code generation, and so on. However, measuring these advanced capabilities across languages is underexplored. To address the disparity, we introduce BenchMAX, a multi-way multilingual evaluation benchmark that allows for fair comparisons of these important abilities across languages. To maintain high quality, three distinct native-speaking annotators independently annotate each sample within all tasks after the data was machine-translated from English into 16 other languages. Additionally, we present a novel translation challenge stemming from dataset construction. Extensive experiments on BenchMAX reveal varying effectiveness of core capabilities across languages, highlighting performance gaps that cannot be bridged by simply scaling up model size. BenchMAX serves as a comprehensive multilingual evaluation platform, providing a promising test bed to promote the development of multilingual language models. The dataset and code are publicly accessible. |
| title | BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.07346 |