Benchmarking LLMs' Mathematical Reasoning with Unseen Random Variables Questions
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913986909306880 |
|---|---|
| author | Hong, Zijin Wu, Hao Dong, Su Dong, Junnan Xiao, Yilin Zhang, Yujing Wang, Zhu Huang, Feiran Li, Linyi Yang, Hongxia Huang, Xiao |
| author_facet | Hong, Zijin Wu, Hao Dong, Su Dong, Junnan Xiao, Yilin Zhang, Yujing Wang, Zhu Huang, Feiran Li, Linyi Yang, Hongxia Huang, Xiao |
| contents | Recent studies have raised significant concerns regarding the reliability of current mathematics benchmarks, highlighting issues such as simplistic design and potential data contamination. Consequently, developing a reliable benchmark that effectively evaluates large language models' (LLMs) genuine capabilities in mathematical reasoning remains a critical challenge. To address these concerns, we propose RV-Bench, a novel evaluation methodology for Benchmarking LLMs with Random Variables in mathematical reasoning. Specifically, we build question-generating functions to produce random variable questions (RVQs), whose background content mirrors original benchmark problems, but with randomized variable combinations, rendering them "unseen" to LLMs. Models must completely understand the inherent question pattern to correctly answer RVQs with diverse variable combinations. Thus, an LLM's genuine reasoning capability is reflected through its accuracy and robustness on RV-Bench. We conducted extensive experiments on over 30 representative LLMs across more than 1,000 RVQs. Our findings propose that LLMs exhibit a proficiency imbalance between encountered and ``unseen'' data distributions. Furthermore, RV-Bench reveals that proficiency generalization across similar mathematical reasoning tasks is limited, but we verified it can still be effectively elicited through test-time scaling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_11790 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Benchmarking LLMs' Mathematical Reasoning with Unseen Random Variables Questions Hong, Zijin Wu, Hao Dong, Su Dong, Junnan Xiao, Yilin Zhang, Yujing Wang, Zhu Huang, Feiran Li, Linyi Yang, Hongxia Huang, Xiao Computation and Language Artificial Intelligence Recent studies have raised significant concerns regarding the reliability of current mathematics benchmarks, highlighting issues such as simplistic design and potential data contamination. Consequently, developing a reliable benchmark that effectively evaluates large language models' (LLMs) genuine capabilities in mathematical reasoning remains a critical challenge. To address these concerns, we propose RV-Bench, a novel evaluation methodology for Benchmarking LLMs with Random Variables in mathematical reasoning. Specifically, we build question-generating functions to produce random variable questions (RVQs), whose background content mirrors original benchmark problems, but with randomized variable combinations, rendering them "unseen" to LLMs. Models must completely understand the inherent question pattern to correctly answer RVQs with diverse variable combinations. Thus, an LLM's genuine reasoning capability is reflected through its accuracy and robustness on RV-Bench. We conducted extensive experiments on over 30 representative LLMs across more than 1,000 RVQs. Our findings propose that LLMs exhibit a proficiency imbalance between encountered and ``unseen'' data distributions. Furthermore, RV-Bench reveals that proficiency generalization across similar mathematical reasoning tasks is limited, but we verified it can still be effectively elicited through test-time scaling. |
| title | Benchmarking LLMs' Mathematical Reasoning with Unseen Random Variables Questions |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2501.11790 |