PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917054020321280 |
|---|---|
| author | Wang, Yiming Zhang, Pei Tang, Jialong Wei, Haoran Yang, Baosong Wang, Rui Sun, Chenshu Sun, Feitong Zhang, Jiran Wu, Junxuan Cang, Qiqian Zhang, Yichang Huang, Fei Lin, Junyang Huang, Fei Zhou, Jingren |
| author_facet | Wang, Yiming Zhang, Pei Tang, Jialong Wei, Haoran Yang, Baosong Wang, Rui Sun, Chenshu Sun, Feitong Zhang, Jiran Wu, Junxuan Cang, Qiqian Zhang, Yichang Huang, Fei Lin, Junyang Huang, Fei Zhou, Jingren |
| contents | In this paper, we introduce PolyMath, a multilingual mathematical reasoning benchmark covering 18 languages and 4 easy-to-hard difficulty levels. Our benchmark ensures difficulty comprehensiveness, language diversity, and high-quality translation, making it a highly discriminative multilingual mathematical benchmark in the era of reasoning LLMs. We conduct a comprehensive evaluation for advanced LLMs and find that even Qwen-3-235B-A22B-Thinking and Gemini-2.5-pro, achieve only 54.6 and 52.2 benchmark scores, with about 40% accuracy under the highest level From a language perspective, our benchmark reveals several key challenges of LLMs in multilingual reasoning: (1) Reasoning performance varies widely across languages for current LLMs; (2) Input-output language consistency is low in reasoning LLMs and may be correlated with performance; (3) The thinking length differs significantly by language for current LLMs. Additionally, we demonstrate that controlling the output language in the instructions has the potential to affect reasoning performance, especially for some low-resource languages, suggesting a promising direction for improving multilingual capabilities in LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_18428 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts Wang, Yiming Zhang, Pei Tang, Jialong Wei, Haoran Yang, Baosong Wang, Rui Sun, Chenshu Sun, Feitong Zhang, Jiran Wu, Junxuan Cang, Qiqian Zhang, Yichang Huang, Fei Lin, Junyang Huang, Fei Zhou, Jingren Computation and Language In this paper, we introduce PolyMath, a multilingual mathematical reasoning benchmark covering 18 languages and 4 easy-to-hard difficulty levels. Our benchmark ensures difficulty comprehensiveness, language diversity, and high-quality translation, making it a highly discriminative multilingual mathematical benchmark in the era of reasoning LLMs. We conduct a comprehensive evaluation for advanced LLMs and find that even Qwen-3-235B-A22B-Thinking and Gemini-2.5-pro, achieve only 54.6 and 52.2 benchmark scores, with about 40% accuracy under the highest level From a language perspective, our benchmark reveals several key challenges of LLMs in multilingual reasoning: (1) Reasoning performance varies widely across languages for current LLMs; (2) Input-output language consistency is low in reasoning LLMs and may be correlated with performance; (3) The thinking length differs significantly by language for current LLMs. Additionally, we demonstrate that controlling the output language in the instructions has the potential to affect reasoning performance, especially for some low-resource languages, suggesting a promising direction for improving multilingual capabilities in LLMs. |
| title | PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2504.18428 |