ETR: Entropy Trend Reward for Efficient Chain-of-Thought Reasoning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917387782062080 |
|---|---|
| author | Xiong, Xuan Liu, Huan Gu, Li Chi, Zhixiang Qiu, Yue Yu, Yuanhao Wang, Yang |
| author_facet | Xiong, Xuan Liu, Huan Gu, Li Chi, Zhixiang Qiu, Yue Yu, Yuanhao Wang, Yang |
| contents | Chain-of-thought (CoT) reasoning improves large language model performance on complex tasks, but often produces excessively long and inefficient reasoning traces. Existing methods shorten CoTs using length penalties or global entropy reduction, implicitly assuming that low uncertainty is desirable throughout reasoning. We show instead that reasoning efficiency is governed by the trajectory of uncertainty. CoTs with dominant downward entropy trends are substantially shorter. Motivated by this insight, we propose Entropy Trend Reward (ETR), a trajectory-aware objective that encourages progressive uncertainty reduction while allowing limited local exploration. We integrate ETR into Group Relative Policy Optimization (GRPO) and evaluate it across multiple reasoning models and challenging benchmarks. ETR consistently achieves a superior accuracy-efficiency tradeoff, improving DeepSeek-R1-Distill-7B by 9.9% in accuracy while reducing CoT length by 67% across four benchmarks. Code is available at https://github.com/Xuan1030/ETR |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_05355 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | ETR: Entropy Trend Reward for Efficient Chain-of-Thought Reasoning Xiong, Xuan Liu, Huan Gu, Li Chi, Zhixiang Qiu, Yue Yu, Yuanhao Wang, Yang Artificial Intelligence Computation and Language Chain-of-thought (CoT) reasoning improves large language model performance on complex tasks, but often produces excessively long and inefficient reasoning traces. Existing methods shorten CoTs using length penalties or global entropy reduction, implicitly assuming that low uncertainty is desirable throughout reasoning. We show instead that reasoning efficiency is governed by the trajectory of uncertainty. CoTs with dominant downward entropy trends are substantially shorter. Motivated by this insight, we propose Entropy Trend Reward (ETR), a trajectory-aware objective that encourages progressive uncertainty reduction while allowing limited local exploration. We integrate ETR into Group Relative Policy Optimization (GRPO) and evaluate it across multiple reasoning models and challenging benchmarks. ETR consistently achieves a superior accuracy-efficiency tradeoff, improving DeepSeek-R1-Distill-7B by 9.9% in accuracy while reducing CoT length by 67% across four benchmarks. Code is available at https://github.com/Xuan1030/ETR |
| title | ETR: Entropy Trend Reward for Efficient Chain-of-Thought Reasoning |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2604.05355 |