Principled RL for Flow Matching Emerges from the Chunk-level Policy Optimization
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911699097878528 |
|---|---|
| author | Luo, Yifu Sun, Haoyuan Hu, Xinhao Du, Penghui Fan, Keyu Li, Bo Du, Sinan Wan, Xu Chen, Zhiyu Xia, Bo Zhang, Tiantian Chang, Yongzhe Yu, Changqian Gai, Kun Wang, Xueqian |
| author_facet | Luo, Yifu Sun, Haoyuan Hu, Xinhao Du, Penghui Fan, Keyu Li, Bo Du, Sinan Wan, Xu Chen, Zhiyu Xia, Bo Zhang, Tiantian Chang, Yongzhe Yu, Changqian Gai, Kun Wang, Xueqian |
| contents | Recent Progress in post-training flow matching for text-to-image (T2I) generation with Group Relative Policy Optimization (GRPO) has demonstrated strong potential. However, it is hindered by a critical limitation: inaccurate advantage attribution. In this work, we argue that aggregating consecutive steps into a coherent `chunk' and shifting the policy optimization paradigm from GRPO's step level to the chunk level can effectively mitigate the negative impact of this issue. Building on this insight, we propose Group Chunking Policy Optimization (GCPO), the first chunk-level reinforcement learning approach for post-training flow matching. Extensive experiments demonstrate that GCPO achieves superior performance on both standard T2I benchmarks and preference alignment, with up to 43% relative gains over GRPO, highlighting the promise of chunk-level policy optimization. The code is available on https://github.com/xingzhejun/GCPO. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_21583 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Principled RL for Flow Matching Emerges from the Chunk-level Policy Optimization Luo, Yifu Sun, Haoyuan Hu, Xinhao Du, Penghui Fan, Keyu Li, Bo Du, Sinan Wan, Xu Chen, Zhiyu Xia, Bo Zhang, Tiantian Chang, Yongzhe Yu, Changqian Gai, Kun Wang, Xueqian Computer Vision and Pattern Recognition Artificial Intelligence Recent Progress in post-training flow matching for text-to-image (T2I) generation with Group Relative Policy Optimization (GRPO) has demonstrated strong potential. However, it is hindered by a critical limitation: inaccurate advantage attribution. In this work, we argue that aggregating consecutive steps into a coherent `chunk' and shifting the policy optimization paradigm from GRPO's step level to the chunk level can effectively mitigate the negative impact of this issue. Building on this insight, we propose Group Chunking Policy Optimization (GCPO), the first chunk-level reinforcement learning approach for post-training flow matching. Extensive experiments demonstrate that GCPO achieves superior performance on both standard T2I benchmarks and preference alignment, with up to 43% relative gains over GRPO, highlighting the promise of chunk-level policy optimization. The code is available on https://github.com/xingzhejun/GCPO. |
| title | Principled RL for Flow Matching Emerges from the Chunk-level Policy Optimization |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2510.21583 |