Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.00296 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917300505935872 |
|---|---|
| author | Li, Xintong Li, Sha Lin, Rongmei Jin, Hongye Li, Linwei Cui, Hejie Zhang, Sarah Chang, Chia-Yuan Cheng, Kewei Fetahu, Besnik Nigam, Priyanka Shang, Jingbo Yin, Bing |
| author_facet | Li, Xintong Li, Sha Lin, Rongmei Jin, Hongye Li, Linwei Cui, Hejie Zhang, Sarah Chang, Chia-Yuan Cheng, Kewei Fetahu, Besnik Nigam, Priyanka Shang, Jingbo Yin, Bing |
| contents | Large reasoning models improve with more test-time computation, but often overthink, producing unnecessarily long chains-of-thought that raise cost without improving accuracy. Prior reinforcement learning approaches typically rely on a single outcome reward with trajectory-level length penalties, which cannot distinguish essential from redundant reasoning steps and therefore yield blunt compression. Although recent work incorporates step-level signals, such as offline pruning, supervised data construction, or verifier-based intermediate rewards, reasoning length is rarely treated as an explicit step-level optimization objective during RL. We propose Step-wise Adaptive Penalization (SWAP), a fine-grained framework that allocates length reduction across steps based on intrinsic contribution. We estimate step importance from the model's on-policy log-probability improvement toward the correct answer, then treat excess length as a penalty mass redistributed to penalize low-importance steps more heavily while preserving high-importance reasoning. We optimize with a unified outcome-process advantage within group-relative policy optimization. Extensive experiments demonstrate that SWAP reduces reasoning length by 64.3% on average while improving accuracy by 5.7% relative to the base model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_00296 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Stepwise Penalization for Length-Efficient Chain-of-Thought Reasoning Li, Xintong Li, Sha Lin, Rongmei Jin, Hongye Li, Linwei Cui, Hejie Zhang, Sarah Chang, Chia-Yuan Cheng, Kewei Fetahu, Besnik Nigam, Priyanka Shang, Jingbo Yin, Bing Computation and Language Artificial Intelligence Machine Learning Large reasoning models improve with more test-time computation, but often overthink, producing unnecessarily long chains-of-thought that raise cost without improving accuracy. Prior reinforcement learning approaches typically rely on a single outcome reward with trajectory-level length penalties, which cannot distinguish essential from redundant reasoning steps and therefore yield blunt compression. Although recent work incorporates step-level signals, such as offline pruning, supervised data construction, or verifier-based intermediate rewards, reasoning length is rarely treated as an explicit step-level optimization objective during RL. We propose Step-wise Adaptive Penalization (SWAP), a fine-grained framework that allocates length reduction across steps based on intrinsic contribution. We estimate step importance from the model's on-policy log-probability improvement toward the correct answer, then treat excess length as a penalty mass redistributed to penalize low-importance steps more heavily while preserving high-importance reasoning. We optimize with a unified outcome-process advantage within group-relative policy optimization. Extensive experiments demonstrate that SWAP reduces reasoning length by 64.3% on average while improving accuracy by 5.7% relative to the base model. |
| title | Stepwise Penalization for Length-Efficient Chain-of-Thought Reasoning |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2603.00296 |