Multi-View Video Diffusion Policy: A 3D Spatio-Temporal-Aware Video Action Model
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908935027425280 |
|---|---|
| author | Li, Peiyan Chen, Yixiang Xu, Yuan Yang, Jiabing Wu, Xiangnan Guo, Jun Sun, Nan Qian, Long Li, Xinghang Xiao, Xin Liu, Jing Liu, Nianfeng Kong, Tao Huang, Yan Wang, Liang Tan, Tieniu |
| author_facet | Li, Peiyan Chen, Yixiang Xu, Yuan Yang, Jiabing Wu, Xiangnan Guo, Jun Sun, Nan Qian, Long Li, Xinghang Xiao, Xin Liu, Jing Liu, Nianfeng Kong, Tao Huang, Yan Wang, Liang Tan, Tieniu |
| contents | Robotic manipulation requires understanding both the 3D spatial structure of the environment and its temporal evolution, yet most existing policies overlook one or both. They typically rely on 2D visual observations and backbones pretrained on static image--text pairs, resulting in high data requirements and limited understanding of environment dynamics. To address this, we introduce MV-VDP, a multi-view video diffusion policy that jointly models the 3D spatio-temporal state of the environment. The core idea is to simultaneously predict multi-view heatmap videos and RGB videos, which 1) align the representation format of video pretraining with action finetuning, and 2) specify not only what actions the robot should take, but also how the environment is expected to evolve in response to those actions. Extensive experiments show that MV-VDP enables data-efficient, robust, generalizable, and interpretable manipulation. With only ten demonstration trajectories and without additional pretraining, MV-VDP successfully performs complex real-world tasks, demonstrates strong robustness across a range of model hyperparameters, generalizes to out-of-distribution settings, and predicts realistic future videos. Experiments on Meta-World and real-world robotic platforms demonstrate that MV-VDP consistently outperforms video-prediction--based, 3D-based, and vision--language--action models, establishing a new state of the art in data-efficient multi-task manipulation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_03181 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Multi-View Video Diffusion Policy: A 3D Spatio-Temporal-Aware Video Action Model Li, Peiyan Chen, Yixiang Xu, Yuan Yang, Jiabing Wu, Xiangnan Guo, Jun Sun, Nan Qian, Long Li, Xinghang Xiao, Xin Liu, Jing Liu, Nianfeng Kong, Tao Huang, Yan Wang, Liang Tan, Tieniu Robotics Computer Vision and Pattern Recognition Robotic manipulation requires understanding both the 3D spatial structure of the environment and its temporal evolution, yet most existing policies overlook one or both. They typically rely on 2D visual observations and backbones pretrained on static image--text pairs, resulting in high data requirements and limited understanding of environment dynamics. To address this, we introduce MV-VDP, a multi-view video diffusion policy that jointly models the 3D spatio-temporal state of the environment. The core idea is to simultaneously predict multi-view heatmap videos and RGB videos, which 1) align the representation format of video pretraining with action finetuning, and 2) specify not only what actions the robot should take, but also how the environment is expected to evolve in response to those actions. Extensive experiments show that MV-VDP enables data-efficient, robust, generalizable, and interpretable manipulation. With only ten demonstration trajectories and without additional pretraining, MV-VDP successfully performs complex real-world tasks, demonstrates strong robustness across a range of model hyperparameters, generalizes to out-of-distribution settings, and predicts realistic future videos. Experiments on Meta-World and real-world robotic platforms demonstrate that MV-VDP consistently outperforms video-prediction--based, 3D-based, and vision--language--action models, establishing a new state of the art in data-efficient multi-task manipulation. |
| title | Multi-View Video Diffusion Policy: A 3D Spatio-Temporal-Aware Video Action Model |
| topic | Robotics Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2604.03181 |