DVPO: Distributional Value Modeling-based Policy Optimization for LLM Post-Training
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913092655382528 |
|---|---|
| author | Zhu, Dingwei Xi, Zhiheng Dou, Shihan Wang, Yuhui Li, Sixian Ye, Junjie Guo, Honglin Liu, Shichun Huang, Chenhao Yang, Yajie Shang, Junlin Jin, Senjie Zhang, Ming Zhang, Jiazheng Huang, Caishuang Zhang, Yunke Wang, Yuran Gui, Tao |
| author_facet | Zhu, Dingwei Xi, Zhiheng Dou, Shihan Wang, Yuhui Li, Sixian Ye, Junjie Guo, Honglin Liu, Shichun Huang, Chenhao Yang, Yajie Shang, Junlin Jin, Senjie Zhang, Ming Zhang, Jiazheng Huang, Caishuang Zhang, Yunke Wang, Yuran Gui, Tao |
| contents | Reinforcement learning (RL) has shown strong performance in LLM post-training, but real-world deployment often involves noisy or incomplete supervision. In such settings, complex and unreliable supervision signals can destabilize training and harm generalization. While existing approaches such as worst-case optimization (e.g., RFQI, CQL) and mean-based methods (e.g., PPO, GRPO) can improve stability, they often overlook generalization and may produce overly conservative policies, leading to uneven performance across diverse real scenarios. To this end, we introduce DVPO (Distributional Value Modeling with Risk-aware Policy Optimization), a new RL framework that combines conditional risk theory with distributional value modeling to better balance robustness and generalization. DVPO learns token-level value distributions to provide fine-grained supervision, and applies an asymmetric risk regularization to shape the distribution tails: it contracts the lower tail to dampen noisy negative deviations, while expanding the upper tail to preserve exploratory diversity. Across extensive experiments and analysis in multi-turn dialogue, math reasoning, and scientific QA, DVPO consistently outperforms PPO, GRPO, and robust Bellman-based PPO under noisy supervision, showing its potential for LLM post-training in the real-world. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_03847 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DVPO: Distributional Value Modeling-based Policy Optimization for LLM Post-Training Zhu, Dingwei Xi, Zhiheng Dou, Shihan Wang, Yuhui Li, Sixian Ye, Junjie Guo, Honglin Liu, Shichun Huang, Chenhao Yang, Yajie Shang, Junlin Jin, Senjie Zhang, Ming Zhang, Jiazheng Huang, Caishuang Zhang, Yunke Wang, Yuran Gui, Tao Machine Learning Artificial Intelligence Reinforcement learning (RL) has shown strong performance in LLM post-training, but real-world deployment often involves noisy or incomplete supervision. In such settings, complex and unreliable supervision signals can destabilize training and harm generalization. While existing approaches such as worst-case optimization (e.g., RFQI, CQL) and mean-based methods (e.g., PPO, GRPO) can improve stability, they often overlook generalization and may produce overly conservative policies, leading to uneven performance across diverse real scenarios. To this end, we introduce DVPO (Distributional Value Modeling with Risk-aware Policy Optimization), a new RL framework that combines conditional risk theory with distributional value modeling to better balance robustness and generalization. DVPO learns token-level value distributions to provide fine-grained supervision, and applies an asymmetric risk regularization to shape the distribution tails: it contracts the lower tail to dampen noisy negative deviations, while expanding the upper tail to preserve exploratory diversity. Across extensive experiments and analysis in multi-turn dialogue, math reasoning, and scientific QA, DVPO consistently outperforms PPO, GRPO, and robust Bellman-based PPO under noisy supervision, showing its potential for LLM post-training in the real-world. |
| title | DVPO: Distributional Value Modeling-based Policy Optimization for LLM Post-Training |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2512.03847 |