AdvantageFlow: Advantage-Weighted Least Squares for RL in Flow Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913161776463872 |
|---|---|
| author | Kveton, Branislav Rao, Anup Mukherjee, Subhojyoti Singh, Krishna Kumar Lai, Viet Dac |
| author_facet | Kveton, Branislav Rao, Anup Mukherjee, Subhojyoti Singh, Krishna Kumar Lai, Viet Dac |
| contents | We introduce AdvantageFlow, a forward-process reinforcement learning algorithm for rectified flow models. Unlike Flow-GRPO, which optimizes the reverse process, we optimize an advantage-weighted forward-process prediction loss. This optimization problem is unstable when advantages are negative and the loss becomes non-convex. We stabilize it by rollout policy regularization, which reduces variance and arises from fitting a local reward-improving target distribution. We evaluate AdvantageFlow on image generation tasks with Stable Diffusion 3.5 Medium. It outperforms both Flow-GRPO and a state-of-the-art forward-process RL baseline based on negative-aware fine-tuning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_26013 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | AdvantageFlow: Advantage-Weighted Least Squares for RL in Flow Models Kveton, Branislav Rao, Anup Mukherjee, Subhojyoti Singh, Krishna Kumar Lai, Viet Dac Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition We introduce AdvantageFlow, a forward-process reinforcement learning algorithm for rectified flow models. Unlike Flow-GRPO, which optimizes the reverse process, we optimize an advantage-weighted forward-process prediction loss. This optimization problem is unstable when advantages are negative and the loss becomes non-convex. We stabilize it by rollout policy regularization, which reduces variance and arises from fitting a local reward-improving target distribution. We evaluate AdvantageFlow on image generation tasks with Stable Diffusion 3.5 Medium. It outperforms both Flow-GRPO and a state-of-the-art forward-process RL baseline based on negative-aware fine-tuning. |
| title | AdvantageFlow: Advantage-Weighted Least Squares for RL in Flow Models |
| topic | Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2605.26013 |