CoMA: Compositional Human Motion Generation with Multi-modal Agents
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916555623759872 |
|---|---|
| author | Sun, Shanlin De Araujo, Gabriel Xu, Jiaqi Zhou, Shenghan Zhang, Hanwen Huang, Ziheng You, Chenyu Xie, Xiaohui |
| author_facet | Sun, Shanlin De Araujo, Gabriel Xu, Jiaqi Zhou, Shenghan Zhang, Hanwen Huang, Ziheng You, Chenyu Xie, Xiaohui |
| contents | 3D human motion generation has seen substantial advancement in recent years. While state-of-the-art approaches have improved performance significantly, they still struggle with complex and detailed motions unseen in training data, largely due to the scarcity of motion datasets and the prohibitive cost of generating new training examples. To address these challenges, we introduce CoMA, an agent-based solution for complex human motion generation, editing, and comprehension. CoMA leverages multiple collaborative agents powered by large language and vision models, alongside a mask transformer-based motion generator featuring body part-specific encoders and codebooks for fine-grained control. Our framework enables generation of both short and long motion sequences with detailed instructions, text-guided motion editing, and self-correction for improved quality. Evaluations on the HumanML3D dataset demonstrate competitive performance against state-of-the-art methods. Additionally, we create a set of context-rich, compositional, and long text prompts, where user studies show our method significantly outperforms existing approaches. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_07320 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CoMA: Compositional Human Motion Generation with Multi-modal Agents Sun, Shanlin De Araujo, Gabriel Xu, Jiaqi Zhou, Shenghan Zhang, Hanwen Huang, Ziheng You, Chenyu Xie, Xiaohui Computer Vision and Pattern Recognition 3D human motion generation has seen substantial advancement in recent years. While state-of-the-art approaches have improved performance significantly, they still struggle with complex and detailed motions unseen in training data, largely due to the scarcity of motion datasets and the prohibitive cost of generating new training examples. To address these challenges, we introduce CoMA, an agent-based solution for complex human motion generation, editing, and comprehension. CoMA leverages multiple collaborative agents powered by large language and vision models, alongside a mask transformer-based motion generator featuring body part-specific encoders and codebooks for fine-grained control. Our framework enables generation of both short and long motion sequences with detailed instructions, text-guided motion editing, and self-correction for improved quality. Evaluations on the HumanML3D dataset demonstrate competitive performance against state-of-the-art methods. Additionally, we create a set of context-rich, compositional, and long text prompts, where user studies show our method significantly outperforms existing approaches. |
| title | CoMA: Compositional Human Motion Generation with Multi-modal Agents |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.07320 |