Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2312.10807 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908914893717504 |
|---|---|
| author | Yao, Xiangtong Zhou, Hongkuan Mees, Oier Meng, Yuan Xiao, Ted Bisk, Yonatan Oh, Jean Johns, Edward Shridhar, Mohit Shah, Dhruv Thomason, Jesse Huang, Kai Chai, Joyce Bing, Zhenshan Knoll, Alois |
| author_facet | Yao, Xiangtong Zhou, Hongkuan Mees, Oier Meng, Yuan Xiao, Ted Bisk, Yonatan Oh, Jean Johns, Edward Shridhar, Mohit Shah, Dhruv Thomason, Jesse Huang, Kai Chai, Joyce Bing, Zhenshan Knoll, Alois |
| contents | Language-conditioned robot manipulation is an emerging field aimed at enabling seamless communication and cooperation between humans and robotic agents by teaching robots to comprehend and execute instructions conveyed in natural language. This interdisciplinary area integrates scene understanding, language processing, and policy learning to bridge the gap between human instructions and robot actions. In this comprehensive survey, we systematically explore recent advancements in language-conditioned robot manipulation. We categorize existing methods based on the primary ways language is integrated into the robot system, namely language for state evaluation, language as a policy condition, language for cognitive planning and reasoning, and language in unified vision-language-action models. Specifically, we further analyze state-of-the-art techniques from five axes of action granularity, data and supervision regimes, system cost and latency, environments and evaluations, and cross-modal task specification. Additionally, we highlight the key debates in the field. Finally, we discuss open challenges and future research directions, focusing on potentially enhancing generalization capabilities and addressing safety issues in language-conditioned robot manipulators. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_10807 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Bridging Language and Action: A Survey of Language-Conditioned Robot Manipulation Yao, Xiangtong Zhou, Hongkuan Mees, Oier Meng, Yuan Xiao, Ted Bisk, Yonatan Oh, Jean Johns, Edward Shridhar, Mohit Shah, Dhruv Thomason, Jesse Huang, Kai Chai, Joyce Bing, Zhenshan Knoll, Alois Robotics Language-conditioned robot manipulation is an emerging field aimed at enabling seamless communication and cooperation between humans and robotic agents by teaching robots to comprehend and execute instructions conveyed in natural language. This interdisciplinary area integrates scene understanding, language processing, and policy learning to bridge the gap between human instructions and robot actions. In this comprehensive survey, we systematically explore recent advancements in language-conditioned robot manipulation. We categorize existing methods based on the primary ways language is integrated into the robot system, namely language for state evaluation, language as a policy condition, language for cognitive planning and reasoning, and language in unified vision-language-action models. Specifically, we further analyze state-of-the-art techniques from five axes of action granularity, data and supervision regimes, system cost and latency, environments and evaluations, and cross-modal task specification. Additionally, we highlight the key debates in the field. Finally, we discuss open challenges and future research directions, focusing on potentially enhancing generalization capabilities and addressing safety issues in language-conditioned robot manipulators. |
| title | Bridging Language and Action: A Survey of Language-Conditioned Robot Manipulation |
| topic | Robotics |
| url | https://arxiv.org/abs/2312.10807 |