Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.10105 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915195500101632 |
|---|---|
| author | Yang, Shu-Xun Wang, Cunxiang Wang, Yidong Gu, Xiaotao Huang, Minlie Tang, Jie |
| author_facet | Yang, Shu-Xun Wang, Cunxiang Wang, Yidong Gu, Xiaotao Huang, Minlie Tang, Jie |
| contents | Evaluating mathematical capabilities is critical for assessing the overall performance of large language models (LLMs). However, existing evaluation methods often focus solely on final answers, resulting in highly inaccurate and uninterpretable evaluation outcomes, as well as their failure to assess proof or open-ended problems. To address these issues, we propose a novel mathematical process evaluation agent based on Tree-of-Error, called StepMathAgent. This agent incorporates four internal core operations: logical step segmentation, step scoring, score aggregation and error tree generation, along with four external extension modules: difficulty calibration, simplicity evaluation, completeness validation and format assessment. Furthermore, we introduce StepMathBench, a benchmark comprising 1,000 step-divided process evaluation instances, derived from 200 high-quality math problems grouped by problem type, subject category and difficulty level. Experiments on StepMathBench show that our proposed StepMathAgent outperforms all state-of-the-art methods, demonstrating human-aligned evaluation preferences and broad applicability to various scenarios. Our data and code are available at https://github.com/SHU-XUN/StepMathAgent. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_10105 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | StepMathAgent: A Step-Wise Agent for Evaluating Mathematical Processes through Tree-of-Error Yang, Shu-Xun Wang, Cunxiang Wang, Yidong Gu, Xiaotao Huang, Minlie Tang, Jie Artificial Intelligence Evaluating mathematical capabilities is critical for assessing the overall performance of large language models (LLMs). However, existing evaluation methods often focus solely on final answers, resulting in highly inaccurate and uninterpretable evaluation outcomes, as well as their failure to assess proof or open-ended problems. To address these issues, we propose a novel mathematical process evaluation agent based on Tree-of-Error, called StepMathAgent. This agent incorporates four internal core operations: logical step segmentation, step scoring, score aggregation and error tree generation, along with four external extension modules: difficulty calibration, simplicity evaluation, completeness validation and format assessment. Furthermore, we introduce StepMathBench, a benchmark comprising 1,000 step-divided process evaluation instances, derived from 200 high-quality math problems grouped by problem type, subject category and difficulty level. Experiments on StepMathBench show that our proposed StepMathAgent outperforms all state-of-the-art methods, demonstrating human-aligned evaluation preferences and broad applicability to various scenarios. Our data and code are available at https://github.com/SHU-XUN/StepMathAgent. |
| title | StepMathAgent: A Step-Wise Agent for Evaluating Mathematical Processes through Tree-of-Error |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2503.10105 |