WildLMa: Long Horizon Loco-Manipulation in the Wild
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913724962439168 |
|---|---|
| author | Qiu, Ri-Zhao Song, Yuchen Peng, Xuanbin Suryadevara, Sai Aneesh Yang, Ge Liu, Minghuan Ji, Mazeyu Jia, Chengzhe Yang, Ruihan Zou, Xueyan Wang, Xiaolong |
| author_facet | Qiu, Ri-Zhao Song, Yuchen Peng, Xuanbin Suryadevara, Sai Aneesh Yang, Ge Liu, Minghuan Ji, Mazeyu Jia, Chengzhe Yang, Ruihan Zou, Xueyan Wang, Xiaolong |
| contents | 'In-the-wild' mobile manipulation aims to deploy robots in diverse real-world environments, which requires the robot to (1) have skills that generalize across object configurations; (2) be capable of long-horizon task execution in diverse environments; and (3) perform complex manipulation beyond pick-and-place. Quadruped robots with manipulators hold promise for extending the workspace and enabling robust locomotion, but existing results do not investigate such a capability. This paper proposes WildLMa with three components to address these issues: (1) adaptation of learned low-level controller for VR-enabled whole-body teleoperation and traversability; (2) WildLMa-Skill -- a library of generalizable visuomotor skills acquired via imitation learning or heuristics and (3) WildLMa-Planner -- an interface of learned skills that allow LLM planners to coordinate skills for long-horizon tasks. We demonstrate the importance of high-quality training data by achieving higher grasping success rate over existing RL baselines using only tens of demonstrations. WildLMa exploits CLIP for language-conditioned imitation learning that empirically generalizes to objects unseen in training demonstrations. Besides extensive quantitative evaluation, we qualitatively demonstrate practical robot applications, such as cleaning up trash in university hallways or outdoor terrains, operating articulated objects, and rearranging items on a bookshelf. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_15131 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | WildLMa: Long Horizon Loco-Manipulation in the Wild Qiu, Ri-Zhao Song, Yuchen Peng, Xuanbin Suryadevara, Sai Aneesh Yang, Ge Liu, Minghuan Ji, Mazeyu Jia, Chengzhe Yang, Ruihan Zou, Xueyan Wang, Xiaolong Robotics Computer Vision and Pattern Recognition Machine Learning 'In-the-wild' mobile manipulation aims to deploy robots in diverse real-world environments, which requires the robot to (1) have skills that generalize across object configurations; (2) be capable of long-horizon task execution in diverse environments; and (3) perform complex manipulation beyond pick-and-place. Quadruped robots with manipulators hold promise for extending the workspace and enabling robust locomotion, but existing results do not investigate such a capability. This paper proposes WildLMa with three components to address these issues: (1) adaptation of learned low-level controller for VR-enabled whole-body teleoperation and traversability; (2) WildLMa-Skill -- a library of generalizable visuomotor skills acquired via imitation learning or heuristics and (3) WildLMa-Planner -- an interface of learned skills that allow LLM planners to coordinate skills for long-horizon tasks. We demonstrate the importance of high-quality training data by achieving higher grasping success rate over existing RL baselines using only tens of demonstrations. WildLMa exploits CLIP for language-conditioned imitation learning that empirically generalizes to objects unseen in training demonstrations. Besides extensive quantitative evaluation, we qualitatively demonstrate practical robot applications, such as cleaning up trash in university hallways or outdoor terrains, operating articulated objects, and rearranging items on a bookshelf. |
| title | WildLMa: Long Horizon Loco-Manipulation in the Wild |
| topic | Robotics Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2411.15131 |