OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916285388947456 |
|---|---|
| author | He, Tairan Luo, Zhengyi He, Xialin Xiao, Wenli Zhang, Chong Zhang, Weinan Kitani, Kris Liu, Changliu Shi, Guanya |
| author_facet | He, Tairan Luo, Zhengyi He, Xialin Xiao, Wenli Zhang, Chong Zhang, Weinan Kitani, Kris Liu, Changliu Shi, Guanya |
| contents | We present OmniH2O (Omni Human-to-Humanoid), a learning-based system for whole-body humanoid teleoperation and autonomy. Using kinematic pose as a universal control interface, OmniH2O enables various ways for a human to control a full-sized humanoid with dexterous hands, including using real-time teleoperation through VR headset, verbal instruction, and RGB camera. OmniH2O also enables full autonomy by learning from teleoperated demonstrations or integrating with frontier models such as GPT-4. OmniH2O demonstrates versatility and dexterity in various real-world whole-body tasks through teleoperation or autonomy, such as playing multiple sports, moving and manipulating objects, and interacting with humans. We develop an RL-based sim-to-real pipeline, which involves large-scale retargeting and augmentation of human motion datasets, learning a real-world deployable policy with sparse sensor input by imitating a privileged teacher policy, and reward designs to enhance robustness and stability. We release the first humanoid whole-body control dataset, OmniH2O-6, containing six everyday tasks, and demonstrate humanoid whole-body skill learning from teleoperated datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_08858 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning He, Tairan Luo, Zhengyi He, Xialin Xiao, Wenli Zhang, Chong Zhang, Weinan Kitani, Kris Liu, Changliu Shi, Guanya Robotics Computer Vision and Pattern Recognition Machine Learning Systems and Control We present OmniH2O (Omni Human-to-Humanoid), a learning-based system for whole-body humanoid teleoperation and autonomy. Using kinematic pose as a universal control interface, OmniH2O enables various ways for a human to control a full-sized humanoid with dexterous hands, including using real-time teleoperation through VR headset, verbal instruction, and RGB camera. OmniH2O also enables full autonomy by learning from teleoperated demonstrations or integrating with frontier models such as GPT-4. OmniH2O demonstrates versatility and dexterity in various real-world whole-body tasks through teleoperation or autonomy, such as playing multiple sports, moving and manipulating objects, and interacting with humans. We develop an RL-based sim-to-real pipeline, which involves large-scale retargeting and augmentation of human motion datasets, learning a real-world deployable policy with sparse sensor input by imitating a privileged teacher policy, and reward designs to enhance robustness and stability. We release the first humanoid whole-body control dataset, OmniH2O-6, containing six everyday tasks, and demonstrate humanoid whole-body skill learning from teleoperated datasets. |
| title | OmniH2O: Universal and Dexterous Human-to-Humanoid Whole-Body Teleoperation and Learning |
| topic | Robotics Computer Vision and Pattern Recognition Machine Learning Systems and Control |
| url | https://arxiv.org/abs/2406.08858 |