RiskPO: Risk-based Policy Optimization via Verifiable Reward for LLM Post-Training
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912621049937920 |
|---|---|
| author | Ren, Tao Jiang, Jinyang Yang, Hui Tian, Wan Zou, Minhao Li, Guanghao Zhang, Zishi Wang, Qinghao Qin, Shentao Zhao, Yanjun Tao, Rui Shao, Hui Peng, Yijie |
| author_facet | Ren, Tao Jiang, Jinyang Yang, Hui Tian, Wan Zou, Minhao Li, Guanghao Zhang, Zishi Wang, Qinghao Qin, Shentao Zhao, Yanjun Tao, Rui Shao, Hui Peng, Yijie |
| contents | Reinforcement learning with verifiable reward has recently emerged as a central paradigm for post-training large language models (LLMs); however, prevailing mean-based methods, such as Group Relative Policy Optimization (GRPO), suffer from entropy collapse and limited reasoning gains. We argue that these issues stem from overemphasizing high-probability output sequences while neglecting rare but informative reasoning paths. To address these challenges, we propose Risk-based Policy Optimization (RiskPO), which substitutes classical mean-based objectives with principled risk measures. Specifically, we introduce a Mixed Value-at-Risk objective that integrates weighted attention over multiple regions of the reward distribution, thereby amplifying gradient signals on challenging instances and preventing overconfident convergence. We further design a bundling scheme that aggregates multiple questions into bundles, thus enriching the feedback signal and yielding more stable and informative training dynamics. Theoretically, we prove that the risk-averse update alleviates entropy collapse and promotes exploration. Numerically, RiskPO achieves consistent and significant improvements in mathematical reasoning, multi-modal reasoning, and code generation benchmarks, surpassing GRPO and its variants on both Pass@1 and Pass@k metrics. Our results demonstrate that risk-based optimization provides a rigorous and effective paradigm for enhancing LLM reasoning capabilities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_00911 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RiskPO: Risk-based Policy Optimization via Verifiable Reward for LLM Post-Training Ren, Tao Jiang, Jinyang Yang, Hui Tian, Wan Zou, Minhao Li, Guanghao Zhang, Zishi Wang, Qinghao Qin, Shentao Zhao, Yanjun Tao, Rui Shao, Hui Peng, Yijie Machine Learning Artificial Intelligence Reinforcement learning with verifiable reward has recently emerged as a central paradigm for post-training large language models (LLMs); however, prevailing mean-based methods, such as Group Relative Policy Optimization (GRPO), suffer from entropy collapse and limited reasoning gains. We argue that these issues stem from overemphasizing high-probability output sequences while neglecting rare but informative reasoning paths. To address these challenges, we propose Risk-based Policy Optimization (RiskPO), which substitutes classical mean-based objectives with principled risk measures. Specifically, we introduce a Mixed Value-at-Risk objective that integrates weighted attention over multiple regions of the reward distribution, thereby amplifying gradient signals on challenging instances and preventing overconfident convergence. We further design a bundling scheme that aggregates multiple questions into bundles, thus enriching the feedback signal and yielding more stable and informative training dynamics. Theoretically, we prove that the risk-averse update alleviates entropy collapse and promotes exploration. Numerically, RiskPO achieves consistent and significant improvements in mathematical reasoning, multi-modal reasoning, and code generation benchmarks, surpassing GRPO and its variants on both Pass@1 and Pass@k metrics. Our results demonstrate that risk-based optimization provides a rigorous and effective paradigm for enhancing LLM reasoning capabilities. |
| title | RiskPO: Risk-based Policy Optimization via Verifiable Reward for LLM Post-Training |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2510.00911 |