From Absolute to Relative: Rethinking Reward Shaping in Group-Based Reinforcement Learning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917235259342848 |
|---|---|
| author | Niu, Wenzhe He, Wei Xie, Zongxia Ou, Jinpeng Fan, Huichuan Ge, Yuchen Sun, Yanru Wang, Ziyin Sun, Yizhao Shi, Chengshun Gao, Jiuchong Hao, Jinghua He, Renqing |
| author_facet | Niu, Wenzhe He, Wei Xie, Zongxia Ou, Jinpeng Fan, Huichuan Ge, Yuchen Sun, Yanru Wang, Ziyin Sun, Yizhao Shi, Chengshun Gao, Jiuchong Hao, Jinghua He, Renqing |
| contents | Reinforcement learning has become a cornerstone for enhancing the reasoning capabilities of Large Language Models, where group-based approaches such as GRPO have emerged as efficient paradigms that optimize policies by leveraging intra-group performance differences. However, these methods typically rely on absolute numerical rewards, introducing intrinsic limitations. In verifiable tasks, identical group evaluations often result in sparse supervision, while in open-ended scenarios, the score range instability of reward models undermines advantage estimation based on group means. To address these limitations, we propose Reinforcement Learning with Relative Rewards (RLRR), a framework that shifts reward shaping from absolute scoring to relative ranking. Complementing this framework, we introduce the Ranking Reward Model, a listwise preference model tailored for group-based optimization to directly generate relative rankings. By transforming raw evaluations into robust relative signals, RLRR effectively mitigates signal sparsity and reward instability. Experimental results demonstrate that RLRR yields consistent performance improvements over standard group-based baselines across reasoning benchmarks and open-ended generation tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_23058 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | From Absolute to Relative: Rethinking Reward Shaping in Group-Based Reinforcement Learning Niu, Wenzhe He, Wei Xie, Zongxia Ou, Jinpeng Fan, Huichuan Ge, Yuchen Sun, Yanru Wang, Ziyin Sun, Yizhao Shi, Chengshun Gao, Jiuchong Hao, Jinghua He, Renqing Machine Learning Reinforcement learning has become a cornerstone for enhancing the reasoning capabilities of Large Language Models, where group-based approaches such as GRPO have emerged as efficient paradigms that optimize policies by leveraging intra-group performance differences. However, these methods typically rely on absolute numerical rewards, introducing intrinsic limitations. In verifiable tasks, identical group evaluations often result in sparse supervision, while in open-ended scenarios, the score range instability of reward models undermines advantage estimation based on group means. To address these limitations, we propose Reinforcement Learning with Relative Rewards (RLRR), a framework that shifts reward shaping from absolute scoring to relative ranking. Complementing this framework, we introduce the Ranking Reward Model, a listwise preference model tailored for group-based optimization to directly generate relative rankings. By transforming raw evaluations into robust relative signals, RLRR effectively mitigates signal sparsity and reward instability. Experimental results demonstrate that RLRR yields consistent performance improvements over standard group-based baselines across reasoning benchmarks and open-ended generation tasks. |
| title | From Absolute to Relative: Rethinking Reward Shaping in Group-Based Reinforcement Learning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2601.23058 |