Beyond Solving Math Quiz: Evaluating the Ability of Large Reasoning Models to Ask for Information
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908490909351936 |
|---|---|
| author | Huang, Youcheng Qin, Bowen Huang, Chen Feng, Duanyu Yang, Xi Lei, Wenqiang |
| author_facet | Huang, Youcheng Qin, Bowen Huang, Chen Feng, Duanyu Yang, Xi Lei, Wenqiang |
| contents | Large Reasoning Models (LRMs) have demonstrated remarkable problem-solving abilities in mathematics, as evaluated by existing benchmarks exclusively on well-defined problems. However, such evaluation setup constitutes a critical gap, since a genuine intelligent agent should not only solve problems (as a math quiz solver), but also be able~to ask for information when the problems lack sufficient information, enabling proactivity in responding users' requests. To bridge such gap, we proposes a new dataset consisting of two types of incomplete problems with diverse contexts. Based on the dataset, our systematical evaluation of LRMs reveals their inability in proactively asking for information. In addition, we uncover the behaviors related to overthinking and hallucination of LRMs, and highlight the potential and challenges of supervised fine-tuning in learning such ability. We hope to provide new insights in developing LRMs with genuine intelligence, rather than just solving problems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_11252 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Beyond Solving Math Quiz: Evaluating the Ability of Large Reasoning Models to Ask for Information Huang, Youcheng Qin, Bowen Huang, Chen Feng, Duanyu Yang, Xi Lei, Wenqiang Artificial Intelligence Computation and Language Information Retrieval Large Reasoning Models (LRMs) have demonstrated remarkable problem-solving abilities in mathematics, as evaluated by existing benchmarks exclusively on well-defined problems. However, such evaluation setup constitutes a critical gap, since a genuine intelligent agent should not only solve problems (as a math quiz solver), but also be able~to ask for information when the problems lack sufficient information, enabling proactivity in responding users' requests. To bridge such gap, we proposes a new dataset consisting of two types of incomplete problems with diverse contexts. Based on the dataset, our systematical evaluation of LRMs reveals their inability in proactively asking for information. In addition, we uncover the behaviors related to overthinking and hallucination of LRMs, and highlight the potential and challenges of supervised fine-tuning in learning such ability. We hope to provide new insights in developing LRMs with genuine intelligence, rather than just solving problems. |
| title | Beyond Solving Math Quiz: Evaluating the Ability of Large Reasoning Models to Ask for Information |
| topic | Artificial Intelligence Computation and Language Information Retrieval |
| url | https://arxiv.org/abs/2508.11252 |