Beware of Reasoning Overconfidence: Pitfalls in the Reasoning Process for Multi-solution Tasks
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909938245173248 |
|---|---|
| author | Guan, Jiannan Chen, Qiguang Qin, Libo Peng, Dengyun Liu, Jinhao Huo, Liangyu Xie, Jian Che, Wanxiang |
| author_facet | Guan, Jiannan Chen, Qiguang Qin, Libo Peng, Dengyun Liu, Jinhao Huo, Liangyu Xie, Jian Che, Wanxiang |
| contents | Large Language Models (LLMs) excel in reasoning tasks requiring a single correct answer, but they perform poorly in multi-solution tasks that require generating comprehensive and diverse answers. We attribute this limitation to \textbf{reasoning overconfidence}: a tendency to express undue certainty in an incomplete solution set. To examine the effect, we introduce \textit{MuSoBench}, a benchmark of multi-solution problems. Experiments show that the conventional short chain-of-thought (Short-CoT) prompting paradigm exhibits pronounced overconfidence, whereas the emerging long chain-of-thought (Long-CoT) approach mitigates it through iterative exploration and self-reflection. We further characterise observable behaviours and influential factors. To probe the underlying cause, we propose the \textbf{cognitive-rigidity hypothesis}, which posits that overconfidence arises when the reasoning process prematurely converges on a narrow set of thought paths. An attention-entropy analysis offers preliminary support for this view. These findings provide tools for assessing the completeness of LLM reasoning and highlight the need to move evaluation beyond single-answer accuracy toward comprehensive exploration. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_01725 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Beware of Reasoning Overconfidence: Pitfalls in the Reasoning Process for Multi-solution Tasks Guan, Jiannan Chen, Qiguang Qin, Libo Peng, Dengyun Liu, Jinhao Huo, Liangyu Xie, Jian Che, Wanxiang Computation and Language Large Language Models (LLMs) excel in reasoning tasks requiring a single correct answer, but they perform poorly in multi-solution tasks that require generating comprehensive and diverse answers. We attribute this limitation to \textbf{reasoning overconfidence}: a tendency to express undue certainty in an incomplete solution set. To examine the effect, we introduce \textit{MuSoBench}, a benchmark of multi-solution problems. Experiments show that the conventional short chain-of-thought (Short-CoT) prompting paradigm exhibits pronounced overconfidence, whereas the emerging long chain-of-thought (Long-CoT) approach mitigates it through iterative exploration and self-reflection. We further characterise observable behaviours and influential factors. To probe the underlying cause, we propose the \textbf{cognitive-rigidity hypothesis}, which posits that overconfidence arises when the reasoning process prematurely converges on a narrow set of thought paths. An attention-entropy analysis offers preliminary support for this view. These findings provide tools for assessing the completeness of LLM reasoning and highlight the need to move evaluation beyond single-answer accuracy toward comprehensive exploration. |
| title | Beware of Reasoning Overconfidence: Pitfalls in the Reasoning Process for Multi-solution Tasks |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2512.01725 |