Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.25889 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911406535737344 |
|---|---|
| author | Chen, Kang Liu, Zhihao Zhang, Tonghe Guo, Zhen Xu, Si Lin, Hao Zang, Hongzhi Li, Xiang Zhang, Quanlu Yu, Zhaofei Fan, Guoliang Huang, Tiejun Wang, Yu Yu, Chao |
| author_facet | Chen, Kang Liu, Zhihao Zhang, Tonghe Guo, Zhen Xu, Si Lin, Hao Zang, Hongzhi Li, Xiang Zhang, Quanlu Yu, Zhaofei Fan, Guoliang Huang, Tiejun Wang, Yu Yu, Chao |
| contents | Vision-Language-Action (VLA) models enable robots to understand and perform complex tasks from multimodal input. Although recent work explores using reinforcement learning (RL) to automate the laborious data collection process in scaling supervised fine-tuning (SFT), applying RL to large-scale flow-based VLAs (\eg, $π_0$, $π_{0.5}$) remains challenging due to intractable action log-likelihoods raised from flow matching. We address this challenge with $π_{\texttt{RL}}$, featuring two technical approaches: (1) \textbf{Flow-Noise} models the denoising process as a discrete-time MDP with a learnable noise network for exact log-likelihood computation. (2) \textbf{Flow-SDE} integrates denoising with agent-environment interaction, formulating a two-layer MDP that employs ODE-to-SDE conversion for efficient RL exploration. We evaluate $π_{\texttt{RL}}$ across various benchmarks, with experiments demonstrating that RL yields significant performance improvements in both in-distribution and out-of-distribution settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_25889 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | $π_\texttt{RL}$: Online RL Fine-tuning for Flow-based Vision-Language-Action Models Chen, Kang Liu, Zhihao Zhang, Tonghe Guo, Zhen Xu, Si Lin, Hao Zang, Hongzhi Li, Xiang Zhang, Quanlu Yu, Zhaofei Fan, Guoliang Huang, Tiejun Wang, Yu Yu, Chao Machine Learning Vision-Language-Action (VLA) models enable robots to understand and perform complex tasks from multimodal input. Although recent work explores using reinforcement learning (RL) to automate the laborious data collection process in scaling supervised fine-tuning (SFT), applying RL to large-scale flow-based VLAs (\eg, $π_0$, $π_{0.5}$) remains challenging due to intractable action log-likelihoods raised from flow matching. We address this challenge with $π_{\texttt{RL}}$, featuring two technical approaches: (1) \textbf{Flow-Noise} models the denoising process as a discrete-time MDP with a learnable noise network for exact log-likelihood computation. (2) \textbf{Flow-SDE} integrates denoising with agent-environment interaction, formulating a two-layer MDP that employs ODE-to-SDE conversion for efficient RL exploration. We evaluate $π_{\texttt{RL}}$ across various benchmarks, with experiments demonstrating that RL yields significant performance improvements in both in-distribution and out-of-distribution settings. |
| title | $π_\texttt{RL}$: Online RL Fine-tuning for Flow-based Vision-Language-Action Models |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2510.25889 |