DFPO: Scaling Value Modeling via Distributional Flow towards Robust and Generalizable LLM Post-Training
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915982838071296 |
|---|---|
| author | Zhu, Dingwei Xi, Zhiheng Dou, Shihan Li, Jiahan Huang, Chenhao Ye, Junjie Li, Sixian Chai, Mingxu Wang, Yuhui Yang, Yajie Zhang, Ming Zhang, Jiazheng Liu, Shichun Huang, Caishuang Zhang, Yunke Wang, Yuran Gui, Tao Qiu, Xipeng Zhang, Qi Huang, Xuanjing |
| author_facet | Zhu, Dingwei Xi, Zhiheng Dou, Shihan Li, Jiahan Huang, Chenhao Ye, Junjie Li, Sixian Chai, Mingxu Wang, Yuhui Yang, Yajie Zhang, Ming Zhang, Jiazheng Liu, Shichun Huang, Caishuang Zhang, Yunke Wang, Yuran Gui, Tao Qiu, Xipeng Zhang, Qi Huang, Xuanjing |
| contents | Training reinforcement learning (RL) systems in real-world environments remains challenging due to noisy supervision and poor out-of-domain (OOD) generalization, especially in LLM post-training. Recent distributional RL methods improve robustness by modeling values with multiple quantile points, but they still learn each quantile independently as a scalar. This results in rough-grained value representations that lack fine-grained conditioning on state information, struggling under complex and OOD conditions. We propose DFPO (Distributional Value Flow Policy Optimization with Conditional Risk and Consistency Control), a robust distributional RL framework that models values as continuous flows across time steps. By scaling value modeling through learning of a value flow field instead of isolated quantile predictions, DFPO captures richer state information for more accurate advantage estimation. To stabilize training under noisy feedback, DFPO further integrates conditional risk control and consistency constraints along value flow trajectories. Experiments on dialogue, math reasoning, and scientific tasks show that DFPO outperforms PPO, FlowRL, and other robust baselines under noisy supervision, achieving improved training stability and generalization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_05890 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | DFPO: Scaling Value Modeling via Distributional Flow towards Robust and Generalizable LLM Post-Training Zhu, Dingwei Xi, Zhiheng Dou, Shihan Li, Jiahan Huang, Chenhao Ye, Junjie Li, Sixian Chai, Mingxu Wang, Yuhui Yang, Yajie Zhang, Ming Zhang, Jiazheng Liu, Shichun Huang, Caishuang Zhang, Yunke Wang, Yuran Gui, Tao Qiu, Xipeng Zhang, Qi Huang, Xuanjing Machine Learning Computation and Language Training reinforcement learning (RL) systems in real-world environments remains challenging due to noisy supervision and poor out-of-domain (OOD) generalization, especially in LLM post-training. Recent distributional RL methods improve robustness by modeling values with multiple quantile points, but they still learn each quantile independently as a scalar. This results in rough-grained value representations that lack fine-grained conditioning on state information, struggling under complex and OOD conditions. We propose DFPO (Distributional Value Flow Policy Optimization with Conditional Risk and Consistency Control), a robust distributional RL framework that models values as continuous flows across time steps. By scaling value modeling through learning of a value flow field instead of isolated quantile predictions, DFPO captures richer state information for more accurate advantage estimation. To stabilize training under noisy feedback, DFPO further integrates conditional risk control and consistency constraints along value flow trajectories. Experiments on dialogue, math reasoning, and scientific tasks show that DFPO outperforms PPO, FlowRL, and other robust baselines under noisy supervision, achieving improved training stability and generalization. |
| title | DFPO: Scaling Value Modeling via Distributional Flow towards Robust and Generalizable LLM Post-Training |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2602.05890 |