Stepwise Perplexity-Guided Refinement for Efficient Chain-of-Thought Reasoning in Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929719958568960 |
|---|---|
| author | Cui, Yingqian He, Pengfei Zeng, Jingying Liu, Hui Tang, Xianfeng Dai, Zhenwei Han, Yan Luo, Chen Huang, Jing Li, Zhen Wang, Suhang Xing, Yue Tang, Jiliang He, Qi |
| author_facet | Cui, Yingqian He, Pengfei Zeng, Jingying Liu, Hui Tang, Xianfeng Dai, Zhenwei Han, Yan Luo, Chen Huang, Jing Li, Zhen Wang, Suhang Xing, Yue Tang, Jiliang He, Qi |
| contents | Chain-of-Thought (CoT) reasoning, which breaks down complex tasks into intermediate reasoning steps, has significantly enhanced the performance of large language models (LLMs) on challenging tasks. However, the detailed reasoning process in CoT often incurs long generation times and high computational costs, partly due to the inclusion of unnecessary steps. To address this, we propose a method to identify critical reasoning steps using perplexity as a measure of their importance: a step is deemed critical if its removal causes a significant increase in perplexity. Our method enables models to focus solely on generating these critical steps. This can be achieved through two approaches: refining demonstration examples in few-shot CoT or fine-tuning the model using selected examples that include only critical steps. Comprehensive experiments validate the effectiveness of our method, which achieves a better balance between the reasoning accuracy and efficiency of CoT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_13260 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Stepwise Perplexity-Guided Refinement for Efficient Chain-of-Thought Reasoning in Large Language Models Cui, Yingqian He, Pengfei Zeng, Jingying Liu, Hui Tang, Xianfeng Dai, Zhenwei Han, Yan Luo, Chen Huang, Jing Li, Zhen Wang, Suhang Xing, Yue Tang, Jiliang He, Qi Computation and Language Artificial Intelligence Machine Learning Chain-of-Thought (CoT) reasoning, which breaks down complex tasks into intermediate reasoning steps, has significantly enhanced the performance of large language models (LLMs) on challenging tasks. However, the detailed reasoning process in CoT often incurs long generation times and high computational costs, partly due to the inclusion of unnecessary steps. To address this, we propose a method to identify critical reasoning steps using perplexity as a measure of their importance: a step is deemed critical if its removal causes a significant increase in perplexity. Our method enables models to focus solely on generating these critical steps. This can be achieved through two approaches: refining demonstration examples in few-shot CoT or fine-tuning the model using selected examples that include only critical steps. Comprehensive experiments validate the effectiveness of our method, which achieves a better balance between the reasoning accuracy and efficiency of CoT. |
| title | Stepwise Perplexity-Guided Refinement for Efficient Chain-of-Thought Reasoning in Large Language Models |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2502.13260 |