LLaRA: Supercharging Robot Learning Data for Vision-Language Policy
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910805417525248 |
|---|---|
| author | Li, Xiang Mata, Cristina Park, Jongwoo Kahatapitiya, Kumara Jang, Yoo Sung Shang, Jinghuan Ranasinghe, Kanchana Burgert, Ryan Cai, Mu Lee, Yong Jae Ryoo, Michael S. |
| author_facet | Li, Xiang Mata, Cristina Park, Jongwoo Kahatapitiya, Kumara Jang, Yoo Sung Shang, Jinghuan Ranasinghe, Kanchana Burgert, Ryan Cai, Mu Lee, Yong Jae Ryoo, Michael S. |
| contents | Vision Language Models (VLMs) have recently been leveraged to generate robotic actions, forming Vision-Language-Action (VLA) models. However, directly adapting a pretrained VLM for robotic control remains challenging, particularly when constrained by a limited number of robot demonstrations. In this work, we introduce LLaRA: Large Language and Robotics Assistant, a framework that formulates robot action policy as visuo-textual conversations and enables an efficient transfer of a pretrained VLM into a powerful VLA, motivated by the success of visual instruction tuning in Computer Vision. First, we present an automated pipeline to generate conversation-style instruction tuning data for robots from existing behavior cloning datasets, aligning robotic actions with image pixel coordinates. Further, we enhance this dataset in a self-supervised manner by defining six auxiliary tasks, without requiring any additional action annotations. We show that a VLM finetuned with a limited amount of such datasets can produce meaningful action decisions for robotic control. Through experiments across multiple simulated and real-world tasks, we demonstrate that LLaRA achieves state-of-the-art performance while preserving the generalization capabilities of large language models. The code, datasets, and pretrained models are available at https://github.com/LostXine/LLaRA. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_20095 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LLaRA: Supercharging Robot Learning Data for Vision-Language Policy Li, Xiang Mata, Cristina Park, Jongwoo Kahatapitiya, Kumara Jang, Yoo Sung Shang, Jinghuan Ranasinghe, Kanchana Burgert, Ryan Cai, Mu Lee, Yong Jae Ryoo, Michael S. Robotics Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Machine Learning Vision Language Models (VLMs) have recently been leveraged to generate robotic actions, forming Vision-Language-Action (VLA) models. However, directly adapting a pretrained VLM for robotic control remains challenging, particularly when constrained by a limited number of robot demonstrations. In this work, we introduce LLaRA: Large Language and Robotics Assistant, a framework that formulates robot action policy as visuo-textual conversations and enables an efficient transfer of a pretrained VLM into a powerful VLA, motivated by the success of visual instruction tuning in Computer Vision. First, we present an automated pipeline to generate conversation-style instruction tuning data for robots from existing behavior cloning datasets, aligning robotic actions with image pixel coordinates. Further, we enhance this dataset in a self-supervised manner by defining six auxiliary tasks, without requiring any additional action annotations. We show that a VLM finetuned with a limited amount of such datasets can produce meaningful action decisions for robotic control. Through experiments across multiple simulated and real-world tasks, we demonstrate that LLaRA achieves state-of-the-art performance while preserving the generalization capabilities of large language models. The code, datasets, and pretrained models are available at https://github.com/LostXine/LLaRA. |
| title | LLaRA: Supercharging Robot Learning Data for Vision-Language Policy |
| topic | Robotics Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2406.20095 |