MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929712792600576 |
|---|---|
| author | Huang, Kaixuan Guo, Jiacheng Li, Zihao Ji, Xiang Ge, Jiawei Li, Wenzhe Guo, Yingqing Cai, Tianle Yuan, Hui Wang, Runzhe Wu, Yue Yin, Ming Tang, Shange Huang, Yangsibo Jin, Chi Chen, Xinyun Zhang, Chiyuan Wang, Mengdi |
| author_facet | Huang, Kaixuan Guo, Jiacheng Li, Zihao Ji, Xiang Ge, Jiawei Li, Wenzhe Guo, Yingqing Cai, Tianle Yuan, Hui Wang, Runzhe Wu, Yue Yin, Ming Tang, Shange Huang, Yangsibo Jin, Chi Chen, Xinyun Zhang, Chiyuan Wang, Mengdi |
| contents | Large language models have demonstrated impressive performance on challenging mathematical reasoning tasks, which has triggered the discussion of whether the performance is achieved by true reasoning capability or memorization. To investigate this question, prior work has constructed mathematical benchmarks when questions undergo simple perturbations -- modifications that still preserve the underlying reasoning patterns of the solutions. However, no work has explored hard perturbations, which fundamentally change the nature of the problem so that the original solution steps do not apply. To bridge the gap, we construct MATH-P-Simple and MATH-P-Hard via simple perturbation and hard perturbation, respectively. Each consists of 279 perturbed math problems derived from level-5 (hardest) problems in the MATH dataset (Hendrycksmath et. al., 2021). We observe significant performance drops on MATH-P-Hard across various models, including o1-mini (-16.49%) and gemini-2.0-flash-thinking (-12.9%). We also raise concerns about a novel form of memorization where models blindly apply learned problem-solving skills without assessing their applicability to modified contexts. This issue is amplified when using original problems for in-context learning. We call for research efforts to address this challenge, which is critical for developing more robust and reliable reasoning models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_06453 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations Huang, Kaixuan Guo, Jiacheng Li, Zihao Ji, Xiang Ge, Jiawei Li, Wenzhe Guo, Yingqing Cai, Tianle Yuan, Hui Wang, Runzhe Wu, Yue Yin, Ming Tang, Shange Huang, Yangsibo Jin, Chi Chen, Xinyun Zhang, Chiyuan Wang, Mengdi Machine Learning Artificial Intelligence Computation and Language Large language models have demonstrated impressive performance on challenging mathematical reasoning tasks, which has triggered the discussion of whether the performance is achieved by true reasoning capability or memorization. To investigate this question, prior work has constructed mathematical benchmarks when questions undergo simple perturbations -- modifications that still preserve the underlying reasoning patterns of the solutions. However, no work has explored hard perturbations, which fundamentally change the nature of the problem so that the original solution steps do not apply. To bridge the gap, we construct MATH-P-Simple and MATH-P-Hard via simple perturbation and hard perturbation, respectively. Each consists of 279 perturbed math problems derived from level-5 (hardest) problems in the MATH dataset (Hendrycksmath et. al., 2021). We observe significant performance drops on MATH-P-Hard across various models, including o1-mini (-16.49%) and gemini-2.0-flash-thinking (-12.9%). We also raise concerns about a novel form of memorization where models blindly apply learned problem-solving skills without assessing their applicability to modified contexts. This issue is amplified when using original problems for in-context learning. We call for research efforts to address this challenge, which is critical for developing more robust and reliable reasoning models. |
| title | MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2502.06453 |