MTR-Bench: A Comprehensive Benchmark for Multi-Turn Reasoning Evaluation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913151411290112 |
|---|---|
| author | Li, Xiaoyuan Bao, Keqin Ma, Yubo Li, Moxin Wang, Wenjie Men, Rui Zhang, Yichang Feng, Fuli Liu, Dayiheng |
| author_facet | Li, Xiaoyuan Bao, Keqin Ma, Yubo Li, Moxin Wang, Wenjie Men, Rui Zhang, Yichang Feng, Fuli Liu, Dayiheng |
| contents | Recent advances in Large Language Models (LLMs) have shown promising results in complex reasoning tasks. However, current evaluations predominantly focus on single-turn reasoning scenarios, leaving interactive tasks largely unexplored. We attribute it to the absence of comprehensive datasets and scalable automatic evaluation protocols. To fill these gaps, we present MTR-Bench for LLMs' Multi-Turn Reasoning evaluation. Comprising 4 classes, 40 tasks, and 3600 instances, MTR-Bench covers diverse reasoning capabilities, fine-grained difficulty granularity, and necessitates multi-turn interactions with the environments. Moreover, MTR-Bench features fully-automated framework spanning both dataset constructions and model evaluations, which enables scalable assessment without human interventions. Extensive experiments reveal that even the cutting-edge reasoning models fall short of multi-turn, interactive reasoning tasks. And the further analysis upon these results brings valuable insights for future research in interactive AI systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_17123 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MTR-Bench: A Comprehensive Benchmark for Multi-Turn Reasoning Evaluation Li, Xiaoyuan Bao, Keqin Ma, Yubo Li, Moxin Wang, Wenjie Men, Rui Zhang, Yichang Feng, Fuli Liu, Dayiheng Computation and Language Recent advances in Large Language Models (LLMs) have shown promising results in complex reasoning tasks. However, current evaluations predominantly focus on single-turn reasoning scenarios, leaving interactive tasks largely unexplored. We attribute it to the absence of comprehensive datasets and scalable automatic evaluation protocols. To fill these gaps, we present MTR-Bench for LLMs' Multi-Turn Reasoning evaluation. Comprising 4 classes, 40 tasks, and 3600 instances, MTR-Bench covers diverse reasoning capabilities, fine-grained difficulty granularity, and necessitates multi-turn interactions with the environments. Moreover, MTR-Bench features fully-automated framework spanning both dataset constructions and model evaluations, which enables scalable assessment without human interventions. Extensive experiments reveal that even the cutting-edge reasoning models fall short of multi-turn, interactive reasoning tasks. And the further analysis upon these results brings valuable insights for future research in interactive AI systems. |
| title | MTR-Bench: A Comprehensive Benchmark for Multi-Turn Reasoning Evaluation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.17123 |