ORV: 4D Occupancy-centric Robot Video Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917096755036160 |
|---|---|
| author | Yang, Xiuyu Li, Bohan Xu, Shaocong Wang, Nan Ye, Chongjie Chen, Zhaoxi Qin, Minghan Ding, Yikang Zhu, Zheng Jin, Xin Zhao, Hang Zhao, Hao |
| author_facet | Yang, Xiuyu Li, Bohan Xu, Shaocong Wang, Nan Ye, Chongjie Chen, Zhaoxi Qin, Minghan Ding, Yikang Zhu, Zheng Jin, Xin Zhao, Hang Zhao, Hao |
| contents | Recent embodied intelligence suffers from data scarcity, while conventional simulators lack visual realism. Controllable video generation is emerging as a promising data engine, yet current action-conditioned methods still fall short: generated videos are limited in fidelity and temporal consistency, poorly aligned with controls, and often constrained to singleview settings. We attribute these issues to the representational gap between sparse control inputs and dense pixel outputs. Thus, we introduce ORV, a 4D occupancy-centric framework for robot video generation that couples action priors with occupancy-derived visual priors. Concretely, we align chunked 7-DoF actions with video latents via an Action-Expert AdaLN modulation, and inject 2D renderings of 4D semantic occupancy into the generation process as soft guidance. Meanwhile, a central obstacle is the lack of occupancy data for embodied scenarios; we therefore curate ORV-Data, a large-scale, high-quality 4D semantic occupancy dataset of robot manipulation. Across BridgeV2, DROID, and RT-1, ORV improves video generation quality and controllability, achieving 18.8% lower FVD than state of the art, +3.5% success rate on visual planning, and +6.4% success rate on policy learning. Beyond singleview generation, ORV natively supports multiview consistent synthesis and enables simulation-to-real transfer despite significant domain gaps. Code, models, and data are at: https://orangesodahub.github.io/ORV |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_03079 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ORV: 4D Occupancy-centric Robot Video Generation Yang, Xiuyu Li, Bohan Xu, Shaocong Wang, Nan Ye, Chongjie Chen, Zhaoxi Qin, Minghan Ding, Yikang Zhu, Zheng Jin, Xin Zhao, Hang Zhao, Hao Computer Vision and Pattern Recognition Recent embodied intelligence suffers from data scarcity, while conventional simulators lack visual realism. Controllable video generation is emerging as a promising data engine, yet current action-conditioned methods still fall short: generated videos are limited in fidelity and temporal consistency, poorly aligned with controls, and often constrained to singleview settings. We attribute these issues to the representational gap between sparse control inputs and dense pixel outputs. Thus, we introduce ORV, a 4D occupancy-centric framework for robot video generation that couples action priors with occupancy-derived visual priors. Concretely, we align chunked 7-DoF actions with video latents via an Action-Expert AdaLN modulation, and inject 2D renderings of 4D semantic occupancy into the generation process as soft guidance. Meanwhile, a central obstacle is the lack of occupancy data for embodied scenarios; we therefore curate ORV-Data, a large-scale, high-quality 4D semantic occupancy dataset of robot manipulation. Across BridgeV2, DROID, and RT-1, ORV improves video generation quality and controllability, achieving 18.8% lower FVD than state of the art, +3.5% success rate on visual planning, and +6.4% success rate on policy learning. Beyond singleview generation, ORV natively supports multiview consistent synthesis and enables simulation-to-real transfer despite significant domain gaps. Code, models, and data are at: https://orangesodahub.github.io/ORV |
| title | ORV: 4D Occupancy-centric Robot Video Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2506.03079 |