Evaluation is All You Need: Strategic Overclaiming of LLM Reasoning Capabilities Through Evaluation Design
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912427194449920 |
|---|---|
| author | Sun, Lin Lin, Weihong Wu, Jinzhu Zhu, Yongfu Jian, Xiaoqi Zhao, Guangxiang Jia, Change Zhang, Linglin Hu, Sai-er Wu, Yuhan Zhang, Xiangzheng |
| author_facet | Sun, Lin Lin, Weihong Wu, Jinzhu Zhu, Yongfu Jian, Xiaoqi Zhao, Guangxiang Jia, Change Zhang, Linglin Hu, Sai-er Wu, Yuhan Zhang, Xiangzheng |
| contents | Reasoning models represented by the Deepseek-R1-Distill series have been widely adopted by the open-source community due to their strong performance in mathematics, science, programming, and other domains. However, our study reveals that their benchmark evaluation results are subject to significant fluctuations caused by various factors. Subtle differences in evaluation conditions can lead to substantial variations in results. Similar phenomena are observed in other open-source inference models fine-tuned based on the Deepseek-R1-Distill series, as well as in the QwQ-32B model, making their claimed performance improvements difficult to reproduce reliably. Therefore, we advocate for the establishment of a more rigorous paradigm for model performance evaluation and present our empirical assessments of the Deepseek-R1-Distill series models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_04734 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Evaluation is All You Need: Strategic Overclaiming of LLM Reasoning Capabilities Through Evaluation Design Sun, Lin Lin, Weihong Wu, Jinzhu Zhu, Yongfu Jian, Xiaoqi Zhao, Guangxiang Jia, Change Zhang, Linglin Hu, Sai-er Wu, Yuhan Zhang, Xiangzheng Artificial Intelligence Computation and Language Machine Learning Reasoning models represented by the Deepseek-R1-Distill series have been widely adopted by the open-source community due to their strong performance in mathematics, science, programming, and other domains. However, our study reveals that their benchmark evaluation results are subject to significant fluctuations caused by various factors. Subtle differences in evaluation conditions can lead to substantial variations in results. Similar phenomena are observed in other open-source inference models fine-tuned based on the Deepseek-R1-Distill series, as well as in the QwQ-32B model, making their claimed performance improvements difficult to reproduce reliably. Therefore, we advocate for the establishment of a more rigorous paradigm for model performance evaluation and present our empirical assessments of the Deepseek-R1-Distill series models. |
| title | Evaluation is All You Need: Strategic Overclaiming of LLM Reasoning Capabilities Through Evaluation Design |
| topic | Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2506.04734 |