Finding Answers in Thought Matters: Revisiting Evaluation on Large Language Models with Reasoning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915558630359040 |
|---|---|
| author | Jo, Hwiyeol Lee, Joosung Lee, Jaehone Lee, Sang-Woo Park, Joonsuk Yoo, Kang Min |
| author_facet | Jo, Hwiyeol Lee, Joosung Lee, Jaehone Lee, Sang-Woo Park, Joonsuk Yoo, Kang Min |
| contents | Evaluating generative models, such as large language models (LLMs), commonly involves question-answering tasks where the final answer is selected based on probability of answer choices. On the other hand, for models requiring reasoning, the method of answer extraction plays a critical role. Our research reveals that the performance of reasoning models and their final answer distributions are highly sensitive to the answer extraction algorithm employed. In order to mitigate this, we propose a basic framework: Answer Regeneration. The method uses an additional model inference, providing the prior input and output prefaced by the prompt "Answer:". The final answer is then selected or extracted from the regenerated output. We show that this extraction-rule-agnostic approach exhibits improved performance and enhanced robustness. Furthermore, we have applied this framework to general math problems and open-ended question answering tasks. Our analysis and this framework could offer a more reliable results for model evaluation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_14773 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Finding Answers in Thought Matters: Revisiting Evaluation on Large Language Models with Reasoning Jo, Hwiyeol Lee, Joosung Lee, Jaehone Lee, Sang-Woo Park, Joonsuk Yoo, Kang Min Computation and Language Artificial Intelligence Evaluating generative models, such as large language models (LLMs), commonly involves question-answering tasks where the final answer is selected based on probability of answer choices. On the other hand, for models requiring reasoning, the method of answer extraction plays a critical role. Our research reveals that the performance of reasoning models and their final answer distributions are highly sensitive to the answer extraction algorithm employed. In order to mitigate this, we propose a basic framework: Answer Regeneration. The method uses an additional model inference, providing the prior input and output prefaced by the prompt "Answer:". The final answer is then selected or extracted from the regenerated output. We show that this extraction-rule-agnostic approach exhibits improved performance and enhanced robustness. Furthermore, we have applied this framework to general math problems and open-ended question answering tasks. Our analysis and this framework could offer a more reliable results for model evaluation. |
| title | Finding Answers in Thought Matters: Revisiting Evaluation on Large Language Models with Reasoning |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2510.14773 |