What Matters in Building Vision-Language-Action Models for Generalist Robots
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917271191945216 |
|---|---|
| author | Li, Xinghang Li, Peiyan Qian, Long Liu, Minghuan Wang, Dong Liu, Jirong Kang, Bingyi Ma, Xiao Wang, Xinlong Guo, Di Kong, Tao Zhang, Hanbo Liu, Huaping |
| author_facet | Li, Xinghang Li, Peiyan Qian, Long Liu, Minghuan Wang, Dong Liu, Jirong Kang, Bingyi Ma, Xiao Wang, Xinlong Guo, Di Kong, Tao Zhang, Hanbo Liu, Huaping |
| contents | To utilize Foundation Vision Language Models (VLMs) for robotic tasks and motion planning, the community has proposed different methods for injecting action components into VLMs and building the Vision-Language-Action models (VLAs). In this work, we disclose the key factors that significantly influence the performance of VLA on robot manipulation problems and focus on answering three essential design choices: which backbone to select, how to formulate the VLA architectures, and when to add cross-embodiment data. The obtained results convince us firmly to explain why we prefer VLA and develop a new family of VLAs, RoboVLMs, which require very few manual designs and achieve a new state-of-the-art performance in three simulation tasks and real-world experiments. Through our extensive experiments, which include over 8 VLM backbones, 4 policy architectures, and over 600 distinct designed experiments, we provide a detailed guidebook for the future design of VLAs. In addition to the study, the highly flexible RoboVLMs framework, which supports easy integrations of new VLMs and free combinations of various design choices, is made public to facilitate future research. We open-source all details, including codes, models, datasets, and toolkits, along with detailed training and evaluation recipes at: robovlms.github.io. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_14058 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | What Matters in Building Vision-Language-Action Models for Generalist Robots Li, Xinghang Li, Peiyan Qian, Long Liu, Minghuan Wang, Dong Liu, Jirong Kang, Bingyi Ma, Xiao Wang, Xinlong Guo, Di Kong, Tao Zhang, Hanbo Liu, Huaping Robotics Computer Vision and Pattern Recognition To utilize Foundation Vision Language Models (VLMs) for robotic tasks and motion planning, the community has proposed different methods for injecting action components into VLMs and building the Vision-Language-Action models (VLAs). In this work, we disclose the key factors that significantly influence the performance of VLA on robot manipulation problems and focus on answering three essential design choices: which backbone to select, how to formulate the VLA architectures, and when to add cross-embodiment data. The obtained results convince us firmly to explain why we prefer VLA and develop a new family of VLAs, RoboVLMs, which require very few manual designs and achieve a new state-of-the-art performance in three simulation tasks and real-world experiments. Through our extensive experiments, which include over 8 VLM backbones, 4 policy architectures, and over 600 distinct designed experiments, we provide a detailed guidebook for the future design of VLAs. In addition to the study, the highly flexible RoboVLMs framework, which supports easy integrations of new VLMs and free combinations of various design choices, is made public to facilitate future research. We open-source all details, including codes, models, datasets, and toolkits, along with detailed training and evaluation recipes at: robovlms.github.io. |
| title | What Matters in Building Vision-Language-Action Models for Generalist Robots |
| topic | Robotics Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.14058 |