Humanoid Locomotion as Next Token Prediction
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909123546710016 |
|---|---|
| author | Radosavovic, Ilija Zhang, Bike Shi, Baifeng Rajasegaran, Jathushan Kamat, Sarthak Darrell, Trevor Sreenath, Koushil Malik, Jitendra |
| author_facet | Radosavovic, Ilija Zhang, Bike Shi, Baifeng Rajasegaran, Jathushan Kamat, Sarthak Darrell, Trevor Sreenath, Koushil Malik, Jitendra |
| contents | We cast real-world humanoid control as a next token prediction problem, akin to predicting the next word in language. Our model is a causal transformer trained via autoregressive prediction of sensorimotor trajectories. To account for the multi-modal nature of the data, we perform prediction in a modality-aligned way, and for each input token predict the next token from the same modality. This general formulation enables us to leverage data with missing modalities, like video trajectories without actions. We train our model on a collection of simulated trajectories coming from prior neural network policies, model-based controllers, motion capture data, and YouTube videos of humans. We show that our model enables a full-sized humanoid to walk in San Francisco zero-shot. Our model can transfer to the real world even when trained on only 27 hours of walking data, and can generalize to commands not seen during training like walking backward. These findings suggest a promising path toward learning challenging real-world control tasks by generative modeling of sensorimotor trajectories. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_19469 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Humanoid Locomotion as Next Token Prediction Radosavovic, Ilija Zhang, Bike Shi, Baifeng Rajasegaran, Jathushan Kamat, Sarthak Darrell, Trevor Sreenath, Koushil Malik, Jitendra Robotics Computer Vision and Pattern Recognition Machine Learning We cast real-world humanoid control as a next token prediction problem, akin to predicting the next word in language. Our model is a causal transformer trained via autoregressive prediction of sensorimotor trajectories. To account for the multi-modal nature of the data, we perform prediction in a modality-aligned way, and for each input token predict the next token from the same modality. This general formulation enables us to leverage data with missing modalities, like video trajectories without actions. We train our model on a collection of simulated trajectories coming from prior neural network policies, model-based controllers, motion capture data, and YouTube videos of humans. We show that our model enables a full-sized humanoid to walk in San Francisco zero-shot. Our model can transfer to the real world even when trained on only 27 hours of walking data, and can generalize to commands not seen during training like walking backward. These findings suggest a promising path toward learning challenging real-world control tasks by generative modeling of sensorimotor trajectories. |
| title | Humanoid Locomotion as Next Token Prediction |
| topic | Robotics Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2402.19469 |