Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.04699 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915432556920832 |
|---|---|
| author | Yadav, Anushka Nalawade, Isha Pillarichety, Srujana Babu, Yashwanth Ghosh, Reshmi Basu, Samyadeep Zhao, Wenlong Nasaeh, Ali Balasubramanian, Sriram Srinivasan, Soundararajan |
| author_facet | Yadav, Anushka Nalawade, Isha Pillarichety, Srujana Babu, Yashwanth Ghosh, Reshmi Basu, Samyadeep Zhao, Wenlong Nasaeh, Ali Balasubramanian, Sriram Srinivasan, Soundararajan |
| contents | The emergence of reasoning models and their integration into practical AI chat bots has led to breakthroughs in solving advanced math, deep search, and extractive question answering problems that requires a complex and multi-step thought process. Yet, a complete understanding of why these models hallucinate more than general purpose language models is missing. In this investigative study, we systematicallyexplore reasoning failures of contemporary language models on multi-hop question answering tasks. We introduce a novel, nuanced error categorization framework that examines failures across three critical dimensions: the diversity and uniqueness of source documents involved ("hops"), completeness in capturing relevant information ("coverage"), and cognitive inefficiency ("overthinking"). Through rigorous hu-man annotation, supported by complementary automated metrics, our exploration uncovers intricate error patterns often hidden by accuracy-centric evaluations. This investigative approach provides deeper insights into the cognitive limitations of current models and offers actionable guidance toward enhancing reasoning fidelity, transparency, and robustness in future language modeling efforts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_04699 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Hop, Skip, and Overthink: Diagnosing Why Reasoning Models Fumble during Multi-Hop Analysis Yadav, Anushka Nalawade, Isha Pillarichety, Srujana Babu, Yashwanth Ghosh, Reshmi Basu, Samyadeep Zhao, Wenlong Nasaeh, Ali Balasubramanian, Sriram Srinivasan, Soundararajan Computation and Language Artificial Intelligence The emergence of reasoning models and their integration into practical AI chat bots has led to breakthroughs in solving advanced math, deep search, and extractive question answering problems that requires a complex and multi-step thought process. Yet, a complete understanding of why these models hallucinate more than general purpose language models is missing. In this investigative study, we systematicallyexplore reasoning failures of contemporary language models on multi-hop question answering tasks. We introduce a novel, nuanced error categorization framework that examines failures across three critical dimensions: the diversity and uniqueness of source documents involved ("hops"), completeness in capturing relevant information ("coverage"), and cognitive inefficiency ("overthinking"). Through rigorous hu-man annotation, supported by complementary automated metrics, our exploration uncovers intricate error patterns often hidden by accuracy-centric evaluations. This investigative approach provides deeper insights into the cognitive limitations of current models and offers actionable guidance toward enhancing reasoning fidelity, transparency, and robustness in future language modeling efforts. |
| title | Hop, Skip, and Overthink: Diagnosing Why Reasoning Models Fumble during Multi-Hop Analysis |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2508.04699 |