Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.12289 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916958568448000 |
|---|---|
| author | Lee, Jinu Hockenmaier, Julia |
| author_facet | Lee, Jinu Hockenmaier, Julia |
| contents | Step-by-step reasoning is widely used to enhance the reasoning ability of large language models (LLMs) in complex problems. Evaluating the quality of reasoning traces is crucial for understanding and improving LLM reasoning. However, existing evaluation practices are highly inconsistent, resulting in fragmented progress across evaluator design and benchmark development. To address this gap, this survey provides a comprehensive overview of step-by-step reasoning evaluation, proposing a taxonomy of evaluation criteria with four top-level categories (factuality, validity, coherence, and utility). Based on the taxonomy, we review different datasets, evaluator implementations, and recent findings, leading to promising directions for future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_12289 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Evaluating Step-by-step Reasoning Traces: A Survey Lee, Jinu Hockenmaier, Julia Computation and Language Step-by-step reasoning is widely used to enhance the reasoning ability of large language models (LLMs) in complex problems. Evaluating the quality of reasoning traces is crucial for understanding and improving LLM reasoning. However, existing evaluation practices are highly inconsistent, resulting in fragmented progress across evaluator design and benchmark development. To address this gap, this survey provides a comprehensive overview of step-by-step reasoning evaluation, proposing a taxonomy of evaluation criteria with four top-level categories (factuality, validity, coherence, and utility). Based on the taxonomy, we review different datasets, evaluator implementations, and recent findings, leading to promising directions for future research. |
| title | Evaluating Step-by-step Reasoning Traces: A Survey |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.12289 |