Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916416206143488 |
|---|---|
| author | Hu, Yafei Xie, Quanting Jain, Vidhi Francis, Jonathan Patrikar, Jay Keetha, Nikhil Kim, Seungchan Xie, Yaqi Zhang, Tianyi Fang, Hao-Shu Zhao, Shibo Omidshafiei, Shayegan Kim, Dong-Ki Agha-mohammadi, Ali-akbar Sycara, Katia Johnson-Roberson, Matthew Batra, Dhruv Wang, Xiaolong Scherer, Sebastian Wang, Chen Kira, Zsolt Xia, Fei Bisk, Yonatan |
| author_facet | Hu, Yafei Xie, Quanting Jain, Vidhi Francis, Jonathan Patrikar, Jay Keetha, Nikhil Kim, Seungchan Xie, Yaqi Zhang, Tianyi Fang, Hao-Shu Zhao, Shibo Omidshafiei, Shayegan Kim, Dong-Ki Agha-mohammadi, Ali-akbar Sycara, Katia Johnson-Roberson, Matthew Batra, Dhruv Wang, Xiaolong Scherer, Sebastian Wang, Chen Kira, Zsolt Xia, Fei Bisk, Yonatan |
| contents | Building general-purpose robots that operate seamlessly in any environment, with any object, and utilizing various skills to complete diverse tasks has been a long-standing goal in Artificial Intelligence. However, as a community, we have been constraining most robotic systems by designing them for specific tasks, training them on specific datasets, and deploying them within specific environments. These systems require extensively-labeled data and task-specific models. When deployed in real-world scenarios, such systems face several generalization issues and struggle to remain robust to distribution shifts. Motivated by the impressive open-set performance and content generation capabilities of web-scale, large-capacity pre-trained models (i.e., foundation models) in research fields such as Natural Language Processing (NLP) and Computer Vision (CV), we devote this survey to exploring (i) how these existing foundation models from NLP and CV can be applied to the field of general-purpose robotics, and also exploring (ii) what a robotics-specific foundation model would look like. We begin by providing a generalized formulation of how foundation models are used in robotics, and the fundamental barriers to making generalist robots universally applicable. Next, we establish a taxonomy to discuss current work exploring ways to leverage existing foundation models for robotics and develop ones catered to robotics. Finally, we discuss key challenges and promising future directions in using foundation models for enabling general-purpose robotic systems. We encourage readers to view our living GitHub repository 2 of resources, including papers reviewed in this survey, as well as related projects and repositories for developing foundation models for robotics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_08782 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis Hu, Yafei Xie, Quanting Jain, Vidhi Francis, Jonathan Patrikar, Jay Keetha, Nikhil Kim, Seungchan Xie, Yaqi Zhang, Tianyi Fang, Hao-Shu Zhao, Shibo Omidshafiei, Shayegan Kim, Dong-Ki Agha-mohammadi, Ali-akbar Sycara, Katia Johnson-Roberson, Matthew Batra, Dhruv Wang, Xiaolong Scherer, Sebastian Wang, Chen Kira, Zsolt Xia, Fei Bisk, Yonatan Robotics Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning Building general-purpose robots that operate seamlessly in any environment, with any object, and utilizing various skills to complete diverse tasks has been a long-standing goal in Artificial Intelligence. However, as a community, we have been constraining most robotic systems by designing them for specific tasks, training them on specific datasets, and deploying them within specific environments. These systems require extensively-labeled data and task-specific models. When deployed in real-world scenarios, such systems face several generalization issues and struggle to remain robust to distribution shifts. Motivated by the impressive open-set performance and content generation capabilities of web-scale, large-capacity pre-trained models (i.e., foundation models) in research fields such as Natural Language Processing (NLP) and Computer Vision (CV), we devote this survey to exploring (i) how these existing foundation models from NLP and CV can be applied to the field of general-purpose robotics, and also exploring (ii) what a robotics-specific foundation model would look like. We begin by providing a generalized formulation of how foundation models are used in robotics, and the fundamental barriers to making generalist robots universally applicable. Next, we establish a taxonomy to discuss current work exploring ways to leverage existing foundation models for robotics and develop ones catered to robotics. Finally, we discuss key challenges and promising future directions in using foundation models for enabling general-purpose robotic systems. We encourage readers to view our living GitHub repository 2 of resources, including papers reviewed in this survey, as well as related projects and repositories for developing foundation models for robotics. |
| title | Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis |
| topic | Robotics Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2312.08782 |