Vision-Language Foundation Models as Effective Robot Imitators
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910318218706944 |
|---|---|
| author | Li, Xinghang Liu, Minghuan Zhang, Hanbo Yu, Cunjun Xu, Jie Wu, Hongtao Cheang, Chilam Jing, Ya Zhang, Weinan Liu, Huaping Li, Hang Kong, Tao |
| author_facet | Li, Xinghang Liu, Minghuan Zhang, Hanbo Yu, Cunjun Xu, Jie Wu, Hongtao Cheang, Chilam Jing, Ya Zhang, Weinan Liu, Huaping Li, Hang Kong, Tao |
| contents | Recent progress in vision language foundation models has shown their ability to understand multimodal data and resolve complicated vision language tasks, including robotics manipulation. We seek a straightforward way of making use of existing vision-language models (VLMs) with simple fine-tuning on robotics data. To this end, we derive a simple and novel vision-language manipulation framework, dubbed RoboFlamingo, built upon the open-source VLMs, OpenFlamingo. Unlike prior works, RoboFlamingo utilizes pre-trained VLMs for single-step vision-language comprehension, models sequential history information with an explicit policy head, and is slightly fine-tuned by imitation learning only on language-conditioned manipulation datasets. Such a decomposition provides RoboFlamingo the flexibility for open-loop control and deployment on low-performance platforms. By exceeding the state-of-the-art performance with a large margin on the tested benchmark, we show RoboFlamingo can be an effective and competitive alternative to adapt VLMs to robot control. Our extensive experimental results also reveal several interesting conclusions regarding the behavior of different pre-trained VLMs on manipulation tasks. We believe RoboFlamingo has the potential to be a cost-effective and easy-to-use solution for robotics manipulation, empowering everyone with the ability to fine-tune their own robotics policy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_01378 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Vision-Language Foundation Models as Effective Robot Imitators Li, Xinghang Liu, Minghuan Zhang, Hanbo Yu, Cunjun Xu, Jie Wu, Hongtao Cheang, Chilam Jing, Ya Zhang, Weinan Liu, Huaping Li, Hang Kong, Tao Robotics Artificial Intelligence Machine Learning Recent progress in vision language foundation models has shown their ability to understand multimodal data and resolve complicated vision language tasks, including robotics manipulation. We seek a straightforward way of making use of existing vision-language models (VLMs) with simple fine-tuning on robotics data. To this end, we derive a simple and novel vision-language manipulation framework, dubbed RoboFlamingo, built upon the open-source VLMs, OpenFlamingo. Unlike prior works, RoboFlamingo utilizes pre-trained VLMs for single-step vision-language comprehension, models sequential history information with an explicit policy head, and is slightly fine-tuned by imitation learning only on language-conditioned manipulation datasets. Such a decomposition provides RoboFlamingo the flexibility for open-loop control and deployment on low-performance platforms. By exceeding the state-of-the-art performance with a large margin on the tested benchmark, we show RoboFlamingo can be an effective and competitive alternative to adapt VLMs to robot control. Our extensive experimental results also reveal several interesting conclusions regarding the behavior of different pre-trained VLMs on manipulation tasks. We believe RoboFlamingo has the potential to be a cost-effective and easy-to-use solution for robotics manipulation, empowering everyone with the ability to fine-tune their own robotics policy. |
| title | Vision-Language Foundation Models as Effective Robot Imitators |
| topic | Robotics Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2311.01378 |