Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.10535 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908878960066560 |
|---|---|
| author | Li, Zichao Lou, Jie Dong, Fangchen Fan, Zhiyuan Ren, Mengjie Lin, Hongyu Han, Xianpei Zhang, Debing Sun, Le Lu, Yaojie Yu, Xing |
| author_facet | Li, Zichao Lou, Jie Dong, Fangchen Fan, Zhiyuan Ren, Mengjie Lin, Hongyu Han, Xianpei Zhang, Debing Sun, Le Lu, Yaojie Yu, Xing |
| contents | Reinforcement learning significantly enhances LLM capabilities but suffers from a critical issue: length inflation, where models adopt verbosity or inefficient reasoning to maximize rewards. Prior approaches struggle to address this challenge in a general and lossless manner, primarily because additive penalties introduce a compensatory effect that creates optimization shortcuts, while heuristic gating strategies lack generality beyond binary feedback. To bridge this gap, we present Group Relative Reward Rescaling (GR$^3$), which reframes length control as a multiplicative rescaling paradigm, effectively establishing a generalized, continuous, and reward-dependent gating mechanism. To further ensure lossless optimization, we incorporate group-relative regularization and advantage-aware calibration, which dynamically adapt length budgets to instance difficulty and preserve the advantage signal of high-quality trajectories. Empirically, across both RLHF and RLVR settings, GR$^3$~maintains training dynamics and downstream performance comparable to standard GRPO while significantly mitigating length inflation, outperforming state-of-the-art length-regularized baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_10535 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning Li, Zichao Lou, Jie Dong, Fangchen Fan, Zhiyuan Ren, Mengjie Lin, Hongyu Han, Xianpei Zhang, Debing Sun, Le Lu, Yaojie Yu, Xing Machine Learning Computation and Language Reinforcement learning significantly enhances LLM capabilities but suffers from a critical issue: length inflation, where models adopt verbosity or inefficient reasoning to maximize rewards. Prior approaches struggle to address this challenge in a general and lossless manner, primarily because additive penalties introduce a compensatory effect that creates optimization shortcuts, while heuristic gating strategies lack generality beyond binary feedback. To bridge this gap, we present Group Relative Reward Rescaling (GR$^3$), which reframes length control as a multiplicative rescaling paradigm, effectively establishing a generalized, continuous, and reward-dependent gating mechanism. To further ensure lossless optimization, we incorporate group-relative regularization and advantage-aware calibration, which dynamically adapt length budgets to instance difficulty and preserve the advantage signal of high-quality trajectories. Empirically, across both RLHF and RLVR settings, GR$^3$~maintains training dynamics and downstream performance comparable to standard GRPO while significantly mitigating length inflation, outperforming state-of-the-art length-regularized baselines. |
| title | Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2603.10535 |