AnySkill: Learning Open-Vocabulary Physical Skill for Interactive Agents
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911803583234048 |
|---|---|
| author | Cui, Jieming Liu, Tengyu Liu, Nian Yang, Yaodong Zhu, Yixin Huang, Siyuan |
| author_facet | Cui, Jieming Liu, Tengyu Liu, Nian Yang, Yaodong Zhu, Yixin Huang, Siyuan |
| contents | Traditional approaches in physics-based motion generation, centered around imitation learning and reward shaping, often struggle to adapt to new scenarios. To tackle this limitation, we propose AnySkill, a novel hierarchical method that learns physically plausible interactions following open-vocabulary instructions. Our approach begins by developing a set of atomic actions via a low-level controller trained via imitation learning. Upon receiving an open-vocabulary textual instruction, AnySkill employs a high-level policy that selects and integrates these atomic actions to maximize the CLIP similarity between the agent's rendered images and the text. An important feature of our method is the use of image-based rewards for the high-level policy, which allows the agent to learn interactions with objects without manual reward engineering. We demonstrate AnySkill's capability to generate realistic and natural motion sequences in response to unseen instructions of varying lengths, marking it the first method capable of open-vocabulary physical skill learning for interactive humanoid agents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_12835 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | AnySkill: Learning Open-Vocabulary Physical Skill for Interactive Agents Cui, Jieming Liu, Tengyu Liu, Nian Yang, Yaodong Zhu, Yixin Huang, Siyuan Computer Vision and Pattern Recognition Robotics Traditional approaches in physics-based motion generation, centered around imitation learning and reward shaping, often struggle to adapt to new scenarios. To tackle this limitation, we propose AnySkill, a novel hierarchical method that learns physically plausible interactions following open-vocabulary instructions. Our approach begins by developing a set of atomic actions via a low-level controller trained via imitation learning. Upon receiving an open-vocabulary textual instruction, AnySkill employs a high-level policy that selects and integrates these atomic actions to maximize the CLIP similarity between the agent's rendered images and the text. An important feature of our method is the use of image-based rewards for the high-level policy, which allows the agent to learn interactions with objects without manual reward engineering. We demonstrate AnySkill's capability to generate realistic and natural motion sequences in response to unseen instructions of varying lengths, marking it the first method capable of open-vocabulary physical skill learning for interactive humanoid agents. |
| title | AnySkill: Learning Open-Vocabulary Physical Skill for Interactive Agents |
| topic | Computer Vision and Pattern Recognition Robotics |
| url | https://arxiv.org/abs/2403.12835 |