Proc4Gem: Foundation models for physical agency through procedural generation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913730286059520 |
|---|---|
| author | Lin, Yixin Humplik, Jan Huang, Sandy H. Hasenclever, Leonard Romano, Francesco Saliceti, Stefano Zheng, Daniel Chen, Jose Enrique Barros, Catarina Collister, Adrian Young, Matt Dostmohamed, Adil Moran, Ben Caluwaerts, Ken Giustina, Marissa Moore, Joss Connell, Kieran Nori, Francesco Heess, Nicolas Bohez, Steven Byravan, Arunkumar |
| author_facet | Lin, Yixin Humplik, Jan Huang, Sandy H. Hasenclever, Leonard Romano, Francesco Saliceti, Stefano Zheng, Daniel Chen, Jose Enrique Barros, Catarina Collister, Adrian Young, Matt Dostmohamed, Adil Moran, Ben Caluwaerts, Ken Giustina, Marissa Moore, Joss Connell, Kieran Nori, Francesco Heess, Nicolas Bohez, Steven Byravan, Arunkumar |
| contents | In robot learning, it is common to either ignore the environment semantics, focusing on tasks like whole-body control which only require reasoning about robot-environment contacts, or conversely to ignore contact dynamics, focusing on grounding high-level movement in vision and language. In this work, we show that advances in generative modeling, photorealistic rendering, and procedural generation allow us to tackle tasks requiring both. By generating contact-rich trajectories with accurate physics in semantically-diverse simulations, we can distill behaviors into large multimodal models that directly transfer to the real world: a system we call Proc4Gem. Specifically, we show that a foundation model, Gemini, fine-tuned on only simulation data, can be instructed in language to control a quadruped robot to push an object with its body to unseen targets in unseen real-world environments. Our real-world results demonstrate the promise of using simulation to imbue foundation models with physical agency. Videos can be found at our website: https://sites.google.com/view/proc4gem |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_08593 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Proc4Gem: Foundation models for physical agency through procedural generation Lin, Yixin Humplik, Jan Huang, Sandy H. Hasenclever, Leonard Romano, Francesco Saliceti, Stefano Zheng, Daniel Chen, Jose Enrique Barros, Catarina Collister, Adrian Young, Matt Dostmohamed, Adil Moran, Ben Caluwaerts, Ken Giustina, Marissa Moore, Joss Connell, Kieran Nori, Francesco Heess, Nicolas Bohez, Steven Byravan, Arunkumar Robotics In robot learning, it is common to either ignore the environment semantics, focusing on tasks like whole-body control which only require reasoning about robot-environment contacts, or conversely to ignore contact dynamics, focusing on grounding high-level movement in vision and language. In this work, we show that advances in generative modeling, photorealistic rendering, and procedural generation allow us to tackle tasks requiring both. By generating contact-rich trajectories with accurate physics in semantically-diverse simulations, we can distill behaviors into large multimodal models that directly transfer to the real world: a system we call Proc4Gem. Specifically, we show that a foundation model, Gemini, fine-tuned on only simulation data, can be instructed in language to control a quadruped robot to push an object with its body to unseen targets in unseen real-world environments. Our real-world results demonstrate the promise of using simulation to imbue foundation models with physical agency. Videos can be found at our website: https://sites.google.com/view/proc4gem |
| title | Proc4Gem: Foundation models for physical agency through procedural generation |
| topic | Robotics |
| url | https://arxiv.org/abs/2503.08593 |