Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.20607 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911532913262592 |
|---|---|
| author | Zhang, Zhilong Ren, Haoxiang Sun, Yihao Sheng, Yifei Wang, Haonan Lin, Haoxin Wu, Zhichao Bacon, Pierre-Luc Yu, Yang |
| author_facet | Zhang, Zhilong Ren, Haoxiang Sun, Yihao Sheng, Yifei Wang, Haonan Lin, Haoxin Wu, Zhichao Bacon, Pierre-Luc Yu, Yang |
| contents | Vision-Language-Action (VLA) models show strong generalization for robotic control, but finetuning them with reinforcement learning (RL) is constrained by the high cost and safety risks of real-world interaction. Training VLA models in interactive world models avoids these issues but introduces several challenges, including pixel-level world modeling, multi-view consistency, and compounding errors under sparse rewards. Building on recent advances across large multimodal models and model-based RL, we propose VLA-MBPO, a practical framework to tackle these problems in VLA finetuning. Our approach has three key design choices: (i) adapting unified multimodal models (UMMs) for data-efficient world modeling; (ii) an interleaved view decoding mechanism to enforce multi-view consistency; and (iii) chunk-level branched rollout to mitigate error compounding. Theoretical analysis and experiments across simulation and real-world tasks demonstrate that VLA-MBPO significantly improves policy performance and sample efficiency, underscoring its robustness and scalability for real-world robotic deployment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_20607 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Towards Practical World Model-based Reinforcement Learning for Vision-Language-Action Models Zhang, Zhilong Ren, Haoxiang Sun, Yihao Sheng, Yifei Wang, Haonan Lin, Haoxin Wu, Zhichao Bacon, Pierre-Luc Yu, Yang Robotics Machine Learning Vision-Language-Action (VLA) models show strong generalization for robotic control, but finetuning them with reinforcement learning (RL) is constrained by the high cost and safety risks of real-world interaction. Training VLA models in interactive world models avoids these issues but introduces several challenges, including pixel-level world modeling, multi-view consistency, and compounding errors under sparse rewards. Building on recent advances across large multimodal models and model-based RL, we propose VLA-MBPO, a practical framework to tackle these problems in VLA finetuning. Our approach has three key design choices: (i) adapting unified multimodal models (UMMs) for data-efficient world modeling; (ii) an interleaved view decoding mechanism to enforce multi-view consistency; and (iii) chunk-level branched rollout to mitigate error compounding. Theoretical analysis and experiments across simulation and real-world tasks demonstrate that VLA-MBPO significantly improves policy performance and sample efficiency, underscoring its robustness and scalability for real-world robotic deployment. |
| title | Towards Practical World Model-based Reinforcement Learning for Vision-Language-Action Models |
| topic | Robotics Machine Learning |
| url | https://arxiv.org/abs/2603.20607 |