ThinkBench: Dynamic Out-of-Distribution Evaluation for Robust LLM Reasoning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910840972640256 |
|---|---|
| author | Huang, Shulin Yang, Linyi Song, Yan Chen, Shuang Cui, Leyang Wan, Ziyu Zeng, Qingcheng Wen, Ying Shao, Kun Zhang, Weinan Wang, Jun Zhang, Yue |
| author_facet | Huang, Shulin Yang, Linyi Song, Yan Chen, Shuang Cui, Leyang Wan, Ziyu Zeng, Qingcheng Wen, Ying Shao, Kun Zhang, Weinan Wang, Jun Zhang, Yue |
| contents | Evaluating large language models (LLMs) poses significant challenges, particularly due to issues of data contamination and the leakage of correct answers. To address these challenges, we introduce ThinkBench, a novel evaluation framework designed to evaluate LLMs' reasoning capability robustly. ThinkBench proposes a dynamic data generation method for constructing out-of-distribution (OOD) datasets and offers an OOD dataset that contains 2,912 samples drawn from reasoning tasks. ThinkBench unifies the evaluation of reasoning models and non-reasoning models. We evaluate 16 LLMs and 4 PRMs under identical experimental conditions and show that most of the LLMs' performance are far from robust and they face a certain level of data leakage. By dynamically generating OOD datasets, ThinkBench effectively provides a reliable evaluation of LLMs and reduces the impact of data contamination. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_16268 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ThinkBench: Dynamic Out-of-Distribution Evaluation for Robust LLM Reasoning Huang, Shulin Yang, Linyi Song, Yan Chen, Shuang Cui, Leyang Wan, Ziyu Zeng, Qingcheng Wen, Ying Shao, Kun Zhang, Weinan Wang, Jun Zhang, Yue Computation and Language Evaluating large language models (LLMs) poses significant challenges, particularly due to issues of data contamination and the leakage of correct answers. To address these challenges, we introduce ThinkBench, a novel evaluation framework designed to evaluate LLMs' reasoning capability robustly. ThinkBench proposes a dynamic data generation method for constructing out-of-distribution (OOD) datasets and offers an OOD dataset that contains 2,912 samples drawn from reasoning tasks. ThinkBench unifies the evaluation of reasoning models and non-reasoning models. We evaluate 16 LLMs and 4 PRMs under identical experimental conditions and show that most of the LLMs' performance are far from robust and they face a certain level of data leakage. By dynamically generating OOD datasets, ThinkBench effectively provides a reliable evaluation of LLMs and reduces the impact of data contamination. |
| title | ThinkBench: Dynamic Out-of-Distribution Evaluation for Robust LLM Reasoning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.16268 |