Improving Vision-Language-Action Model with Online Reinforcement Learning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909468695986176 |
|---|---|
| author | Guo, Yanjiang Zhang, Jianke Chen, Xiaoyu Ji, Xiang Wang, Yen-Jen Hu, Yucheng Chen, Jianyu |
| author_facet | Guo, Yanjiang Zhang, Jianke Chen, Xiaoyu Ji, Xiang Wang, Yen-Jen Hu, Yucheng Chen, Jianyu |
| contents | Recent studies have successfully integrated large vision-language models (VLMs) into low-level robotic control by supervised fine-tuning (SFT) with expert robotic datasets, resulting in what we term vision-language-action (VLA) models. Although the VLA models are powerful, how to improve these large models during interaction with environments remains an open question. In this paper, we explore how to further improve these VLA models via Reinforcement Learning (RL), a commonly used fine-tuning technique for large models. However, we find that directly applying online RL to large VLA models presents significant challenges, including training instability that severely impacts the performance of large models, and computing burdens that exceed the capabilities of most local machines. To address these challenges, we propose iRe-VLA framework, which iterates between Reinforcement Learning and Supervised Learning to effectively improve VLA models, leveraging the exploratory benefits of RL while maintaining the stability of supervised learning. Experiments in two simulated benchmarks and a real-world manipulation suite validate the effectiveness of our method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_16664 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Improving Vision-Language-Action Model with Online Reinforcement Learning Guo, Yanjiang Zhang, Jianke Chen, Xiaoyu Ji, Xiang Wang, Yen-Jen Hu, Yucheng Chen, Jianyu Robotics Computer Vision and Pattern Recognition Machine Learning Recent studies have successfully integrated large vision-language models (VLMs) into low-level robotic control by supervised fine-tuning (SFT) with expert robotic datasets, resulting in what we term vision-language-action (VLA) models. Although the VLA models are powerful, how to improve these large models during interaction with environments remains an open question. In this paper, we explore how to further improve these VLA models via Reinforcement Learning (RL), a commonly used fine-tuning technique for large models. However, we find that directly applying online RL to large VLA models presents significant challenges, including training instability that severely impacts the performance of large models, and computing burdens that exceed the capabilities of most local machines. To address these challenges, we propose iRe-VLA framework, which iterates between Reinforcement Learning and Supervised Learning to effectively improve VLA models, leveraging the exploratory benefits of RL while maintaining the stability of supervised learning. Experiments in two simulated benchmarks and a real-world manipulation suite validate the effectiveness of our method. |
| title | Improving Vision-Language-Action Model with Online Reinforcement Learning |
| topic | Robotics Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2501.16664 |