DreamGen: Unlocking Generalization in Robot Learning through Video World Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912436824571904 |
|---|---|
| author | Jang, Joel Ye, Seonghyeon Lin, Zongyu Xiang, Jiannan Bjorck, Johan Fang, Yu Hu, Fengyuan Huang, Spencer Kundalia, Kaushil Lin, Yen-Chen Magne, Loic Mandlekar, Ajay Narayan, Avnish Tan, You Liang Wang, Guanzhi Wang, Jing Wang, Qi Xu, Yinzhen Zeng, Xiaohui Zheng, Kaiyuan Zheng, Ruijie Liu, Ming-Yu Zettlemoyer, Luke Fox, Dieter Kautz, Jan Reed, Scott Zhu, Yuke Fan, Linxi |
| author_facet | Jang, Joel Ye, Seonghyeon Lin, Zongyu Xiang, Jiannan Bjorck, Johan Fang, Yu Hu, Fengyuan Huang, Spencer Kundalia, Kaushil Lin, Yen-Chen Magne, Loic Mandlekar, Ajay Narayan, Avnish Tan, You Liang Wang, Guanzhi Wang, Jing Wang, Qi Xu, Yinzhen Zeng, Xiaohui Zheng, Kaiyuan Zheng, Ruijie Liu, Ming-Yu Zettlemoyer, Luke Fox, Dieter Kautz, Jan Reed, Scott Zhu, Yuke Fan, Linxi |
| contents | We introduce DreamGen, a simple yet highly effective 4-stage pipeline for training robot policies that generalize across behaviors and environments through neural trajectories - synthetic robot data generated from video world models. DreamGen leverages state-of-the-art image-to-video generative models, adapting them to the target robot embodiment to produce photorealistic synthetic videos of familiar or novel tasks in diverse environments. Since these models generate only videos, we recover pseudo-action sequences using either a latent action model or an inverse-dynamics model (IDM). Despite its simplicity, DreamGen unlocks strong behavior and environment generalization: a humanoid robot can perform 22 new behaviors in both seen and unseen environments, while requiring teleoperation data from only a single pick-and-place task in one environment. To evaluate the pipeline systematically, we introduce DreamGen Bench, a video generation benchmark that shows a strong correlation between benchmark performance and downstream policy success. Our work establishes a promising new axis for scaling robot learning well beyond manual data collection. Code available at https://github.com/NVIDIA/GR00T-Dreams. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_12705 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DreamGen: Unlocking Generalization in Robot Learning through Video World Models Jang, Joel Ye, Seonghyeon Lin, Zongyu Xiang, Jiannan Bjorck, Johan Fang, Yu Hu, Fengyuan Huang, Spencer Kundalia, Kaushil Lin, Yen-Chen Magne, Loic Mandlekar, Ajay Narayan, Avnish Tan, You Liang Wang, Guanzhi Wang, Jing Wang, Qi Xu, Yinzhen Zeng, Xiaohui Zheng, Kaiyuan Zheng, Ruijie Liu, Ming-Yu Zettlemoyer, Luke Fox, Dieter Kautz, Jan Reed, Scott Zhu, Yuke Fan, Linxi Robotics Artificial Intelligence Machine Learning We introduce DreamGen, a simple yet highly effective 4-stage pipeline for training robot policies that generalize across behaviors and environments through neural trajectories - synthetic robot data generated from video world models. DreamGen leverages state-of-the-art image-to-video generative models, adapting them to the target robot embodiment to produce photorealistic synthetic videos of familiar or novel tasks in diverse environments. Since these models generate only videos, we recover pseudo-action sequences using either a latent action model or an inverse-dynamics model (IDM). Despite its simplicity, DreamGen unlocks strong behavior and environment generalization: a humanoid robot can perform 22 new behaviors in both seen and unseen environments, while requiring teleoperation data from only a single pick-and-place task in one environment. To evaluate the pipeline systematically, we introduce DreamGen Bench, a video generation benchmark that shows a strong correlation between benchmark performance and downstream policy success. Our work establishes a promising new axis for scaling robot learning well beyond manual data collection. Code available at https://github.com/NVIDIA/GR00T-Dreams. |
| title | DreamGen: Unlocking Generalization in Robot Learning through Video World Models |
| topic | Robotics Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2505.12705 |