Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.15677 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918070769942528 |
|---|---|
| author | Jia, Mingxi Huang, Haojie Zhang, Zhewen Wang, Chenghao Zhao, Linfeng Wang, Dian Liu, Jason Xinyu Walters, Robin Platt, Robert Tellex, Stefanie |
| author_facet | Jia, Mingxi Huang, Haojie Zhang, Zhewen Wang, Chenghao Zhao, Linfeng Wang, Dian Liu, Jason Xinyu Walters, Robin Platt, Robert Tellex, Stefanie |
| contents | Controlling robots through natural language is pivotal for enhancing human-robot collaboration and synthesizing complex robot behaviors. Recent works that are trained on large robot datasets show impressive generalization abilities. However, such pretrained methods are (1) often fragile to unseen scenarios, and (2) expensive to adapt to new tasks. This paper introduces Grounded Equivariant Manipulation (GEM), a robust yet efficient approach that leverages pretrained vision-language models with equivariant language mapping for language-conditioned manipulation tasks. Our experiments demonstrate GEM's high sample efficiency and generalization ability across diverse tasks in both simulation and the real world. GEM achieves similar or higher performance with orders of magnitude fewer robot data compared with major data-efficient baselines such as CLIPort and VIMA. Finally, our approach demonstrates greater robustness compared to large VLA model, e.g, OpenVLA, at correctly interpreting natural language commands on unseen objects and poses. Code, data, and training details are available https://saulbatman.github.io/gem_page/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_15677 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Learning Efficient and Robust Language-conditioned Manipulation using Textual-Visual Relevancy and Equivariant Language Mapping Jia, Mingxi Huang, Haojie Zhang, Zhewen Wang, Chenghao Zhao, Linfeng Wang, Dian Liu, Jason Xinyu Walters, Robin Platt, Robert Tellex, Stefanie Robotics Controlling robots through natural language is pivotal for enhancing human-robot collaboration and synthesizing complex robot behaviors. Recent works that are trained on large robot datasets show impressive generalization abilities. However, such pretrained methods are (1) often fragile to unseen scenarios, and (2) expensive to adapt to new tasks. This paper introduces Grounded Equivariant Manipulation (GEM), a robust yet efficient approach that leverages pretrained vision-language models with equivariant language mapping for language-conditioned manipulation tasks. Our experiments demonstrate GEM's high sample efficiency and generalization ability across diverse tasks in both simulation and the real world. GEM achieves similar or higher performance with orders of magnitude fewer robot data compared with major data-efficient baselines such as CLIPort and VIMA. Finally, our approach demonstrates greater robustness compared to large VLA model, e.g, OpenVLA, at correctly interpreting natural language commands on unseen objects and poses. Code, data, and training details are available https://saulbatman.github.io/gem_page/ |
| title | Learning Efficient and Robust Language-conditioned Manipulation using Textual-Visual Relevancy and Equivariant Language Mapping |
| topic | Robotics |
| url | https://arxiv.org/abs/2406.15677 |