HumanVLA: Towards Vision-Language Directed Object Rearrangement by Physical Humanoid
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917835787206656 |
|---|---|
| author | Xu, Xinyu Zhang, Yizheng Li, Yong-Lu Han, Lei Lu, Cewu |
| author_facet | Xu, Xinyu Zhang, Yizheng Li, Yong-Lu Han, Lei Lu, Cewu |
| contents | Physical Human-Scene Interaction (HSI) plays a crucial role in numerous applications.
However, existing HSI techniques are limited to specific object dynamics and privileged information, which prevents the development of more comprehensive applications.
To address this limitation, we introduce HumanVLA for general object rearrangement directed by practical vision and language.
A teacher-student framework is utilized to develop HumanVLA.
A state-based teacher policy is trained first using goal-conditioned reinforcement learning and adversarial motion prior.
Then, it is distilled into a vision-language-action model via behavior cloning.
We propose several key insights to facilitate the large-scale learning process.
To support general object rearrangement by physical humanoid, we introduce a novel Human-in-the-Room dataset encompassing various rearrangement tasks.
Through extensive experiments and analysis, we demonstrate the effectiveness of the proposed approach. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_19972 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | HumanVLA: Towards Vision-Language Directed Object Rearrangement by Physical Humanoid Xu, Xinyu Zhang, Yizheng Li, Yong-Lu Han, Lei Lu, Cewu Robotics Physical Human-Scene Interaction (HSI) plays a crucial role in numerous applications. However, existing HSI techniques are limited to specific object dynamics and privileged information, which prevents the development of more comprehensive applications. To address this limitation, we introduce HumanVLA for general object rearrangement directed by practical vision and language. A teacher-student framework is utilized to develop HumanVLA. A state-based teacher policy is trained first using goal-conditioned reinforcement learning and adversarial motion prior. Then, it is distilled into a vision-language-action model via behavior cloning. We propose several key insights to facilitate the large-scale learning process. To support general object rearrangement by physical humanoid, we introduce a novel Human-in-the-Room dataset encompassing various rearrangement tasks. Through extensive experiments and analysis, we demonstrate the effectiveness of the proposed approach. |
| title | HumanVLA: Towards Vision-Language Directed Object Rearrangement by Physical Humanoid |
| topic | Robotics |
| url | https://arxiv.org/abs/2406.19972 |