Stepwise Credit Assignment for GRPO on Flow-Matching Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908921803833344 |
|---|---|
| author | Savani, Yash Kveton, Branislav Liu, Yuchen Wang, Yilin Shi, Jing Mukherjee, Subhojyoti Vlassis, Nikos Singh, Krishna Kumar |
| author_facet | Savani, Yash Kveton, Branislav Liu, Yuchen Wang, Yilin Shi, Jing Mukherjee, Subhojyoti Vlassis, Nikos Singh, Krishna Kumar |
| contents | Flow-GRPO successfully applies reinforcement learning to flow models, but uses uniform credit assignment across all steps. This ignores the temporal structure of diffusion generation: early steps determine composition and content (low-frequency structure), while late steps resolve details and textures (high-frequency details). Moreover, assigning uniform credit based solely on the final image can inadvertently reward suboptimal intermediate steps, especially when errors are corrected later in the diffusion trajectory. We propose Stepwise-Flow-GRPO, which assigns credit based on each step's reward improvement. By leveraging Tweedie's formula to obtain intermediate reward estimates and introducing gain-based advantages, our method achieves superior sample efficiency and faster convergence. We also introduce a DDIM-inspired SDE that improves reward quality while preserving stochasticity for policy gradients. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_28718 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Stepwise Credit Assignment for GRPO on Flow-Matching Models Savani, Yash Kveton, Branislav Liu, Yuchen Wang, Yilin Shi, Jing Mukherjee, Subhojyoti Vlassis, Nikos Singh, Krishna Kumar Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Flow-GRPO successfully applies reinforcement learning to flow models, but uses uniform credit assignment across all steps. This ignores the temporal structure of diffusion generation: early steps determine composition and content (low-frequency structure), while late steps resolve details and textures (high-frequency details). Moreover, assigning uniform credit based solely on the final image can inadvertently reward suboptimal intermediate steps, especially when errors are corrected later in the diffusion trajectory. We propose Stepwise-Flow-GRPO, which assigns credit based on each step's reward improvement. By leveraging Tweedie's formula to obtain intermediate reward estimates and introducing gain-based advantages, our method achieves superior sample efficiency and faster convergence. We also introduce a DDIM-inspired SDE that improves reward quality while preserving stochasticity for policy gradients. |
| title | Stepwise Credit Assignment for GRPO on Flow-Matching Models |
| topic | Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2603.28718 |