WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RL
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915799500849152 |
|---|---|
| author | Jiang, Zhennan Zhou, Shangqing Jiang, Yutong Huang, Zefang Wei, Mingjie Chen, Yuhui Zhou, Tianxing Guo, Zhen Lin, Hao Zhang, Quanlu Wang, Yu Li, Haoran Yu, Chao Zhao, Dongbin |
| author_facet | Jiang, Zhennan Zhou, Shangqing Jiang, Yutong Huang, Zefang Wei, Mingjie Chen, Yuhui Zhou, Tianxing Guo, Zhen Lin, Hao Zhang, Quanlu Wang, Yu Li, Haoran Yu, Chao Zhao, Dongbin |
| contents | Reinforcement learning (RL) promises to unlock capabilities beyond imitation learning for Vision-Language-Action (VLA) models, but its requirement for massive real-world interaction prevents direct deployment on physical robots. Recent work attempts to use learned world models as simulators for policy optimization, yet closed-loop imagined rollouts inevitably suffer from hallucination and long-horizon error accumulation. Such errors do not merely degrade visual fidelity; they corrupt the optimization signal, encouraging policies to exploit model inaccuracies rather than genuine task progress. We propose WoVR, a reliable world-model-based reinforcement learning framework for post-training VLA policies. Instead of assuming a faithful world model, WoVR explicitly regulates how RL interacts with imperfect imagined dynamics. It improves rollout stability through a controllable action-conditioned video world model, reshapes imagined interaction to reduce effective error depth via Keyframe-Initialized Rollouts, and maintains policy-simulator alignment through World Model-Policy co-evolution. Extensive experiments on LIBERO benchmarks and real-world robotic manipulation demonstrate that WoVR enables stable long-horizon imagined rollouts and effective policy optimization, improving average LIBERO success from 39.95% to 69.2% (+29.3 points) and real-robot success from 61.7% to 91.7% (+30.0 points). These results show that learned world models can serve as practical simulators for reinforcement learning when hallucination is explicitly controlled. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_13977 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RL Jiang, Zhennan Zhou, Shangqing Jiang, Yutong Huang, Zefang Wei, Mingjie Chen, Yuhui Zhou, Tianxing Guo, Zhen Lin, Hao Zhang, Quanlu Wang, Yu Li, Haoran Yu, Chao Zhao, Dongbin Robotics Artificial Intelligence Reinforcement learning (RL) promises to unlock capabilities beyond imitation learning for Vision-Language-Action (VLA) models, but its requirement for massive real-world interaction prevents direct deployment on physical robots. Recent work attempts to use learned world models as simulators for policy optimization, yet closed-loop imagined rollouts inevitably suffer from hallucination and long-horizon error accumulation. Such errors do not merely degrade visual fidelity; they corrupt the optimization signal, encouraging policies to exploit model inaccuracies rather than genuine task progress. We propose WoVR, a reliable world-model-based reinforcement learning framework for post-training VLA policies. Instead of assuming a faithful world model, WoVR explicitly regulates how RL interacts with imperfect imagined dynamics. It improves rollout stability through a controllable action-conditioned video world model, reshapes imagined interaction to reduce effective error depth via Keyframe-Initialized Rollouts, and maintains policy-simulator alignment through World Model-Policy co-evolution. Extensive experiments on LIBERO benchmarks and real-world robotic manipulation demonstrate that WoVR enables stable long-horizon imagined rollouts and effective policy optimization, improving average LIBERO success from 39.95% to 69.2% (+29.3 points) and real-robot success from 61.7% to 91.7% (+30.0 points). These results show that learned world models can serve as practical simulators for reinforcement learning when hallucination is explicitly controlled. |
| title | WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RL |
| topic | Robotics Artificial Intelligence |
| url | https://arxiv.org/abs/2602.13977 |