TLControl: Trajectory and Language Control for Human Motion Synthesis
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909266053431296 |
|---|---|
| author | Wan, Weilin Dou, Zhiyang Komura, Taku Wang, Wenping Jayaraman, Dinesh Liu, Lingjie |
| author_facet | Wan, Weilin Dou, Zhiyang Komura, Taku Wang, Wenping Jayaraman, Dinesh Liu, Lingjie |
| contents | Controllable human motion synthesis is essential for applications in AR/VR, gaming and embodied AI. Existing methods often focus solely on either language or full trajectory control, lacking precision in synthesizing motions aligned with user-specified trajectories, especially for multi-joint control. To address these issues, we present TLControl, a novel method for realistic human motion synthesis, incorporating both low-level Trajectory and high-level Language semantics controls, through the integration of neural-based and optimization-based techniques. Specifically, we begin with training a VQ-VAE for a compact and well-structured latent motion space organized by body parts. We then propose a Masked Trajectories Transformer (MTT) for predicting a motion distribution conditioned on language and trajectory. Once trained, we use MTT to sample initial motion predictions given user-specified partial trajectories and text descriptions as conditioning. Finally, we introduce a test-time optimization to refine these coarse predictions for precise trajectory control, which offers flexibility by allowing users to specify various optimization goals and ensures high runtime efficiency. Comprehensive experiments show that TLControl significantly outperforms the state-of-the-art in trajectory accuracy and time efficiency, making it practical for interactive and high-quality animation generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_17135 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | TLControl: Trajectory and Language Control for Human Motion Synthesis Wan, Weilin Dou, Zhiyang Komura, Taku Wang, Wenping Jayaraman, Dinesh Liu, Lingjie Computer Vision and Pattern Recognition Graphics Controllable human motion synthesis is essential for applications in AR/VR, gaming and embodied AI. Existing methods often focus solely on either language or full trajectory control, lacking precision in synthesizing motions aligned with user-specified trajectories, especially for multi-joint control. To address these issues, we present TLControl, a novel method for realistic human motion synthesis, incorporating both low-level Trajectory and high-level Language semantics controls, through the integration of neural-based and optimization-based techniques. Specifically, we begin with training a VQ-VAE for a compact and well-structured latent motion space organized by body parts. We then propose a Masked Trajectories Transformer (MTT) for predicting a motion distribution conditioned on language and trajectory. Once trained, we use MTT to sample initial motion predictions given user-specified partial trajectories and text descriptions as conditioning. Finally, we introduce a test-time optimization to refine these coarse predictions for precise trajectory control, which offers flexibility by allowing users to specify various optimization goals and ensures high runtime efficiency. Comprehensive experiments show that TLControl significantly outperforms the state-of-the-art in trajectory accuracy and time efficiency, making it practical for interactive and high-quality animation generation. |
| title | TLControl: Trajectory and Language Control for Human Motion Synthesis |
| topic | Computer Vision and Pattern Recognition Graphics |
| url | https://arxiv.org/abs/2311.17135 |