MotionGPT-2: A General-Purpose Motion-Language Model for Motion Generation and Understanding
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914996779220992 |
|---|---|
| author | Wang, Yuan Huang, Di Zhang, Yaqi Ouyang, Wanli Jiao, Jile Feng, Xuetao Zhou, Yan Wan, Pengfei Tang, Shixiang Xu, Dan |
| author_facet | Wang, Yuan Huang, Di Zhang, Yaqi Ouyang, Wanli Jiao, Jile Feng, Xuetao Zhou, Yan Wan, Pengfei Tang, Shixiang Xu, Dan |
| contents | Generating lifelike human motions from descriptive texts has experienced remarkable research focus in the recent years, propelled by the emerging requirements of digital humans.Despite impressive advances, existing approaches are often constrained by limited control modalities, task specificity, and focus solely on body motion representations.In this paper, we present MotionGPT-2, a unified Large Motion-Language Model (LMLM) that addresses these limitations. MotionGPT-2 accommodates multiple motion-relevant tasks and supporting multimodal control conditions through pre-trained Large Language Models (LLMs). It quantizes multimodal inputs-such as text and single-frame poses-into discrete, LLM-interpretable tokens, seamlessly integrating them into the LLM's vocabulary. These tokens are then organized into unified prompts, guiding the LLM to generate motion outputs through a pretraining-then-finetuning paradigm. We also show that the proposed MotionGPT-2 is highly adaptable to the challenging 3D holistic motion generation task, enabled by the innovative motion discretization framework, Part-Aware VQVAE, which ensures fine-grained representations of body and hand movements. Extensive experiments and visualizations validate the effectiveness of our method, demonstrating the adaptability of MotionGPT-2 across motion generation, motion captioning, and generalized motion completion tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_21747 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MotionGPT-2: A General-Purpose Motion-Language Model for Motion Generation and Understanding Wang, Yuan Huang, Di Zhang, Yaqi Ouyang, Wanli Jiao, Jile Feng, Xuetao Zhou, Yan Wan, Pengfei Tang, Shixiang Xu, Dan Computer Vision and Pattern Recognition Generating lifelike human motions from descriptive texts has experienced remarkable research focus in the recent years, propelled by the emerging requirements of digital humans.Despite impressive advances, existing approaches are often constrained by limited control modalities, task specificity, and focus solely on body motion representations.In this paper, we present MotionGPT-2, a unified Large Motion-Language Model (LMLM) that addresses these limitations. MotionGPT-2 accommodates multiple motion-relevant tasks and supporting multimodal control conditions through pre-trained Large Language Models (LLMs). It quantizes multimodal inputs-such as text and single-frame poses-into discrete, LLM-interpretable tokens, seamlessly integrating them into the LLM's vocabulary. These tokens are then organized into unified prompts, guiding the LLM to generate motion outputs through a pretraining-then-finetuning paradigm. We also show that the proposed MotionGPT-2 is highly adaptable to the challenging 3D holistic motion generation task, enabled by the innovative motion discretization framework, Part-Aware VQVAE, which ensures fine-grained representations of body and hand movements. Extensive experiments and visualizations validate the effectiveness of our method, demonstrating the adaptability of MotionGPT-2 across motion generation, motion captioning, and generalized motion completion tasks. |
| title | MotionGPT-2: A General-Purpose Motion-Language Model for Motion Generation and Understanding |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.21747 |