Large Video Planner Enables Generalizable Robot Control
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911660039471104 |
|---|---|
| author | Chen, Boyuan Zhang, Tianyuan Geng, Haoran Zhang, Caiyi Li, Peihao Song, Kiwhan Freeman, William T. Malik, Jitendra Abbeel, Pieter Tedrake, Russ Sitzmann, Vincent Du, Yilun |
| author_facet | Chen, Boyuan Zhang, Tianyuan Geng, Haoran Zhang, Caiyi Li, Peihao Song, Kiwhan Freeman, William T. Malik, Jitendra Abbeel, Pieter Tedrake, Russ Sitzmann, Vincent Du, Yilun |
| contents | General-purpose robots require decision-making models that generalize across diverse tasks and environments. Recent works build robot foundation models by extending multimodal large language models (MLLMs) with action outputs, creating vision-language-action (VLA) systems. These efforts are motivated by the intuition that MLLMs' large-scale language and image pretraining can be effectively transferred to the action output modality. In this work, we explore an alternative paradigm of using large-scale video pretraining as a primary modality for building robot foundation models. Unlike static images and language, videos capture spatio-temporal sequences of states and actions in the physical world that are naturally aligned with robotic behavior. We curate an internet-scale video dataset of human activities and task demonstrations, and train, for the first time at a foundation-model scale, an open video model for generative robotics planning. The model produces zero-shot video plans for novel scenes and tasks, which we post-process to extract executable robot actions. We evaluate task-level generalization through third-party selected tasks in the wild and real-robot experiments, demonstrating successful physical execution. Together, these results show robust instruction following, strong generalization, and real-world feasibility. We release both the model and dataset to support open, reproducible video-based robot learning. Our website is available at https://www.boyuan.space/large-video-planner/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_15840 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Large Video Planner Enables Generalizable Robot Control Chen, Boyuan Zhang, Tianyuan Geng, Haoran Zhang, Caiyi Li, Peihao Song, Kiwhan Freeman, William T. Malik, Jitendra Abbeel, Pieter Tedrake, Russ Sitzmann, Vincent Du, Yilun Robotics Computer Vision and Pattern Recognition General-purpose robots require decision-making models that generalize across diverse tasks and environments. Recent works build robot foundation models by extending multimodal large language models (MLLMs) with action outputs, creating vision-language-action (VLA) systems. These efforts are motivated by the intuition that MLLMs' large-scale language and image pretraining can be effectively transferred to the action output modality. In this work, we explore an alternative paradigm of using large-scale video pretraining as a primary modality for building robot foundation models. Unlike static images and language, videos capture spatio-temporal sequences of states and actions in the physical world that are naturally aligned with robotic behavior. We curate an internet-scale video dataset of human activities and task demonstrations, and train, for the first time at a foundation-model scale, an open video model for generative robotics planning. The model produces zero-shot video plans for novel scenes and tasks, which we post-process to extract executable robot actions. We evaluate task-level generalization through third-party selected tasks in the wild and real-robot experiments, demonstrating successful physical execution. Together, these results show robust instruction following, strong generalization, and real-world feasibility. We release both the model and dataset to support open, reproducible video-based robot learning. Our website is available at https://www.boyuan.space/large-video-planner/. |
| title | Large Video Planner Enables Generalizable Robot Control |
| topic | Robotics Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2512.15840 |