Flow Matching Policy Gradients
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915421045653504 |
|---|---|
| author | McAllister, David Ge, Songwei Yi, Brent Kim, Chung Min Weber, Ethan Choi, Hongsuk Feng, Haiwen Kanazawa, Angjoo |
| author_facet | McAllister, David Ge, Songwei Yi, Brent Kim, Chung Min Weber, Ethan Choi, Hongsuk Feng, Haiwen Kanazawa, Angjoo |
| contents | Flow-based generative models, including diffusion models, excel at modeling continuous distributions in high-dimensional spaces. In this work, we introduce Flow Policy Optimization (FPO), a simple on-policy reinforcement learning algorithm that brings flow matching into the policy gradient framework. FPO casts policy optimization as maximizing an advantage-weighted ratio computed from the conditional flow matching loss, in a manner compatible with the popular PPO-clip framework. It sidesteps the need for exact likelihood computation while preserving the generative capabilities of flow-based models. Unlike prior approaches for diffusion-based reinforcement learning that bind training to a specific sampling method, FPO is agnostic to the choice of diffusion or flow integration at both training and inference time. We show that FPO can train diffusion-style policies from scratch in a variety of continuous control tasks. We find that flow-based models can capture multimodal action distributions and achieve higher performance than Gaussian policies, particularly in under-conditioned settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_21053 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Flow Matching Policy Gradients McAllister, David Ge, Songwei Yi, Brent Kim, Chung Min Weber, Ethan Choi, Hongsuk Feng, Haiwen Kanazawa, Angjoo Machine Learning Robotics Flow-based generative models, including diffusion models, excel at modeling continuous distributions in high-dimensional spaces. In this work, we introduce Flow Policy Optimization (FPO), a simple on-policy reinforcement learning algorithm that brings flow matching into the policy gradient framework. FPO casts policy optimization as maximizing an advantage-weighted ratio computed from the conditional flow matching loss, in a manner compatible with the popular PPO-clip framework. It sidesteps the need for exact likelihood computation while preserving the generative capabilities of flow-based models. Unlike prior approaches for diffusion-based reinforcement learning that bind training to a specific sampling method, FPO is agnostic to the choice of diffusion or flow integration at both training and inference time. We show that FPO can train diffusion-style policies from scratch in a variety of continuous control tasks. We find that flow-based models can capture multimodal action distributions and achieve higher performance than Gaussian policies, particularly in under-conditioned settings. |
| title | Flow Matching Policy Gradients |
| topic | Machine Learning Robotics |
| url | https://arxiv.org/abs/2507.21053 |