Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.03441 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914964703281152 |
|---|---|
| author | Tevet, Guy Raab, Sigal Cohan, Setareh Reda, Daniele Luo, Zhengyi Peng, Xue Bin Bermano, Amit H. van de Panne, Michiel |
| author_facet | Tevet, Guy Raab, Sigal Cohan, Setareh Reda, Daniele Luo, Zhengyi Peng, Xue Bin Bermano, Amit H. van de Panne, Michiel |
| contents | Motion diffusion models and Reinforcement Learning (RL) based control for physics-based simulations have complementary strengths for human motion generation. The former is capable of generating a wide variety of motions, adhering to intuitive control such as text, while the latter offers physically plausible motion and direct interaction with the environment. In this work, we present a method that combines their respective strengths. CLoSD is a text-driven RL physics-based controller, guided by diffusion generation for various tasks. Our key insight is that motion diffusion can serve as an on-the-fly universal planner for a robust RL controller. To this end, CLoSD maintains a closed-loop interaction between two modules -- a Diffusion Planner (DiP), and a tracking controller. DiP is a fast-responding autoregressive diffusion model, controlled by textual prompts and target locations, and the controller is a simple and robust motion imitator that continuously receives motion plans from DiP and provides feedback from the environment. CLoSD is capable of seamlessly performing a sequence of different tasks, including navigation to a goal location, striking an object with a hand or foot as specified in a text prompt, sitting down, and getting up. https://guytevet.github.io/CLoSD-page/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_03441 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character control Tevet, Guy Raab, Sigal Cohan, Setareh Reda, Daniele Luo, Zhengyi Peng, Xue Bin Bermano, Amit H. van de Panne, Michiel Computer Vision and Pattern Recognition Motion diffusion models and Reinforcement Learning (RL) based control for physics-based simulations have complementary strengths for human motion generation. The former is capable of generating a wide variety of motions, adhering to intuitive control such as text, while the latter offers physically plausible motion and direct interaction with the environment. In this work, we present a method that combines their respective strengths. CLoSD is a text-driven RL physics-based controller, guided by diffusion generation for various tasks. Our key insight is that motion diffusion can serve as an on-the-fly universal planner for a robust RL controller. To this end, CLoSD maintains a closed-loop interaction between two modules -- a Diffusion Planner (DiP), and a tracking controller. DiP is a fast-responding autoregressive diffusion model, controlled by textual prompts and target locations, and the controller is a simple and robust motion imitator that continuously receives motion plans from DiP and provides feedback from the environment. CLoSD is capable of seamlessly performing a sequence of different tasks, including navigation to a goal location, striking an object with a hand or foot as specified in a text prompt, sitting down, and getting up. https://guytevet.github.io/CLoSD-page/ |
| title | CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character control |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.03441 |