VRPO: Rethinking Value Modeling for Robust RL Training under Noisy Supervision
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911092176846848 |
|---|---|
| author | Zhu, Dingwei Dou, Shihan Xi, Zhiheng Jin, Senjie Zhang, Guoqiang Zhang, Jiazheng Ye, Junjie Chai, Mingxu Zhou, Enyu Zhang, Ming Huang, Caishuang Zhang, Yunke Wang, Yuran Gui, Tao |
| author_facet | Zhu, Dingwei Dou, Shihan Xi, Zhiheng Jin, Senjie Zhang, Guoqiang Zhang, Jiazheng Ye, Junjie Chai, Mingxu Zhou, Enyu Zhang, Ming Huang, Caishuang Zhang, Yunke Wang, Yuran Gui, Tao |
| contents | Reinforcement Learning from Human Feedback (RLHF) often suffers from noisy or imperfect reward supervision in real-world settings, which undermines policy stability and generalization. Such noise may cause models to lose attention on key words during advantage estimation. While prior work focuses on reward denoising or filtering poor data, it often overlooks the critical role of the value model in policy optimization. In this work, we show that a strong value model is essential for mitigating noise by absorbing unstable signals and enabling more reliable advantage estimation. We propose VRPO, a value-centric framework for robust PPO training under noisy supervision. VRPO combines two core designs: (1) an auxiliary loss guided by entropy and perplexity from a frozen language model, and (2) a variational information bottleneck. These mechanisms enhance the value model's ability to filter out noise and capture key words from the context during advantage estimation, transforming it from a passive predictor into an active regulator of noise. Experiments on math reasoning, science QA, and multi-turn dialogue, under both rule-based and model-based noisy rewards, show that VRPO consistently outperforms PPO and GRPO baselines. Our findings underscore the often-overlooked importance of the value model in RLHF and offer a principled and practical approach to robust policy optimization in noisy real-world environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_03058 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | VRPO: Rethinking Value Modeling for Robust RL Training under Noisy Supervision Zhu, Dingwei Dou, Shihan Xi, Zhiheng Jin, Senjie Zhang, Guoqiang Zhang, Jiazheng Ye, Junjie Chai, Mingxu Zhou, Enyu Zhang, Ming Huang, Caishuang Zhang, Yunke Wang, Yuran Gui, Tao Machine Learning Artificial Intelligence Computation and Language Reinforcement Learning from Human Feedback (RLHF) often suffers from noisy or imperfect reward supervision in real-world settings, which undermines policy stability and generalization. Such noise may cause models to lose attention on key words during advantage estimation. While prior work focuses on reward denoising or filtering poor data, it often overlooks the critical role of the value model in policy optimization. In this work, we show that a strong value model is essential for mitigating noise by absorbing unstable signals and enabling more reliable advantage estimation. We propose VRPO, a value-centric framework for robust PPO training under noisy supervision. VRPO combines two core designs: (1) an auxiliary loss guided by entropy and perplexity from a frozen language model, and (2) a variational information bottleneck. These mechanisms enhance the value model's ability to filter out noise and capture key words from the context during advantage estimation, transforming it from a passive predictor into an active regulator of noise. Experiments on math reasoning, science QA, and multi-turn dialogue, under both rule-based and model-based noisy rewards, show that VRPO consistently outperforms PPO and GRPO baselines. Our findings underscore the often-overlooked importance of the value model in RLHF and offer a principled and practical approach to robust policy optimization in noisy real-world environments. |
| title | VRPO: Rethinking Value Modeling for Robust RL Training under Noisy Supervision |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2508.03058 |