Thinking Economically: A Hierarchical Framework for Adaptive-Complexity Reasoning in LLMs
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918533979439104 |
|---|---|
| author | Gao, Yubo Wu, Haotian Chen, Hong Huang, Junquan Yan, Yibo Li, Jungang Dongfang, Zihao Tao, Sicheng Tan, Puay Siew Zhang, Jie Hu, Xuming |
| author_facet | Gao, Yubo Wu, Haotian Chen, Hong Huang, Junquan Yan, Yibo Li, Jungang Dongfang, Zihao Tao, Sicheng Tan, Puay Siew Zhang, Jie Hu, Xuming |
| contents | Chain-of-Thought (CoT) has significantly enhanced LLM reasoning, yet often incurs substantial computational overhead due to "overthinking": generating excessively long rationales without commensurate accuracy gains. Existing efficiency methods typically apply uniform compression, which overlooks a critical observation that reasoning complexity is heterogeneous at two distinct granularity: across different problems and within individual reasoning steps. This motivates our principle of Thinking Economically: intelligently allocating computational resources based on intrinsic task and step demands rather than pursuing uniform brevity. We propose Hierarchical Adaptive Budgeter (HAB), a training framework that operationalizes this principle through coarse-to-fine budgeting. At the inter-step level, HAB predicts the optimal reasoning depth for each problem. At the intra-step level, HAB learns step-specific token budgeting signals from PPL-derived step comparisons and an adaptive Pareto optimization objective that captures the local quality-efficiency trade-off, while a Fisher Information-based pruner further provides fine-grained training-time guidance, thereby encouraging the generator to internalize more economical reasoning patterns. Experiments on GSM8K and MATH500 show that HAB not only surpasses standard CoT in accuracy but also reduces token usage, achieving a stronger performance-efficiency trade-off than the compared baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2606_01168 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Thinking Economically: A Hierarchical Framework for Adaptive-Complexity Reasoning in LLMs Gao, Yubo Wu, Haotian Chen, Hong Huang, Junquan Yan, Yibo Li, Jungang Dongfang, Zihao Tao, Sicheng Tan, Puay Siew Zhang, Jie Hu, Xuming Computation and Language Chain-of-Thought (CoT) has significantly enhanced LLM reasoning, yet often incurs substantial computational overhead due to "overthinking": generating excessively long rationales without commensurate accuracy gains. Existing efficiency methods typically apply uniform compression, which overlooks a critical observation that reasoning complexity is heterogeneous at two distinct granularity: across different problems and within individual reasoning steps. This motivates our principle of Thinking Economically: intelligently allocating computational resources based on intrinsic task and step demands rather than pursuing uniform brevity. We propose Hierarchical Adaptive Budgeter (HAB), a training framework that operationalizes this principle through coarse-to-fine budgeting. At the inter-step level, HAB predicts the optimal reasoning depth for each problem. At the intra-step level, HAB learns step-specific token budgeting signals from PPL-derived step comparisons and an adaptive Pareto optimization objective that captures the local quality-efficiency trade-off, while a Fisher Information-based pruner further provides fine-grained training-time guidance, thereby encouraging the generator to internalize more economical reasoning patterns. Experiments on GSM8K and MATH500 show that HAB not only surpasses standard CoT in accuracy but also reduces token usage, achieving a stronger performance-efficiency trade-off than the compared baselines. |
| title | Thinking Economically: A Hierarchical Framework for Adaptive-Complexity Reasoning in LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2606.01168 |