Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.09586 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908317163454464 |
|---|---|
| author | Tang, Zuoli Ou, Junjie Hu, Kaiqin Wu, Chunwei Huan, Zhaoxin Fu, Chilin Zhang, Xiaolu Zhou, Jun Li, Chenliang |
| author_facet | Tang, Zuoli Ou, Junjie Hu, Kaiqin Wu, Chunwei Huan, Zhaoxin Fu, Chilin Zhang, Xiaolu Zhou, Jun Li, Chenliang |
| contents | Recent years have witnessed significant progress in large language models' (LLMs) reasoning, which is largely due to the chain-of-thought (CoT) approaches, allowing models to generate intermediate reasoning steps before reaching the final answer. Building on these advances, state-of-the-art LLMs are instruction-tuned to provide long and detailed CoT pathways when responding to reasoning-related questions. However, human beings are naturally cognitive misers and will prompt language models to give rather short responses, thus raising a significant conflict with CoT reasoning. In this paper, we delve into how LLMs' reasoning performance changes when users provide short-path prompts. The results and analysis reveal that language models can reason effectively and robustly without explicit CoT prompts, while under short-path prompting, LLMs' reasoning ability drops significantly and becomes unstable, even on grade-school problems. To address this issue, we propose two approaches: an instruction-guided approach and a fine-tuning approach, both designed to effectively manage the conflict. Experimental results show that both methods achieve high accuracy, providing insights into the trade-off between instruction adherence and reasoning accuracy in current models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_09586 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Short-Path Prompting in LLMs: Analyzing Reasoning Instability and Solutions for Robust Performance Tang, Zuoli Ou, Junjie Hu, Kaiqin Wu, Chunwei Huan, Zhaoxin Fu, Chilin Zhang, Xiaolu Zhou, Jun Li, Chenliang Computation and Language Recent years have witnessed significant progress in large language models' (LLMs) reasoning, which is largely due to the chain-of-thought (CoT) approaches, allowing models to generate intermediate reasoning steps before reaching the final answer. Building on these advances, state-of-the-art LLMs are instruction-tuned to provide long and detailed CoT pathways when responding to reasoning-related questions. However, human beings are naturally cognitive misers and will prompt language models to give rather short responses, thus raising a significant conflict with CoT reasoning. In this paper, we delve into how LLMs' reasoning performance changes when users provide short-path prompts. The results and analysis reveal that language models can reason effectively and robustly without explicit CoT prompts, while under short-path prompting, LLMs' reasoning ability drops significantly and becomes unstable, even on grade-school problems. To address this issue, we propose two approaches: an instruction-guided approach and a fine-tuning approach, both designed to effectively manage the conflict. Experimental results show that both methods achieve high accuracy, providing insights into the trade-off between instruction adherence and reasoning accuracy in current models. |
| title | Short-Path Prompting in LLMs: Analyzing Reasoning Instability and Solutions for Robust Performance |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2504.09586 |