BOTH2Hands: Inferring 3D Hands from Both Text Prompts and Body Dynamics
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912046797291520 |
|---|---|
| author | Zhang, Wenqian Huang, Molin Zhou, Yuxuan Zhang, Juze Yu, Jingyi Wang, Jingya Xu, Lan |
| author_facet | Zhang, Wenqian Huang, Molin Zhou, Yuxuan Zhang, Juze Yu, Jingyi Wang, Jingya Xu, Lan |
| contents | The recently emerging text-to-motion advances have spired numerous attempts for convenient and interactive human motion generation. Yet, existing methods are largely limited to generating body motions only without considering the rich two-hand motions, let alone handling various conditions like body dynamics or texts. To break the data bottleneck, we propose BOTH57M, a novel multi-modal dataset for two-hand motion generation. Our dataset includes accurate motion tracking for the human body and hands and provides pair-wised finger-level hand annotations and body descriptions. We further provide a strong baseline method, BOTH2Hands, for the novel task: generating vivid two-hand motions from both implicit body dynamics and explicit text prompts. We first warm up two parallel body-to-hand and text-to-hand diffusion models and then utilize the cross-attention transformer for motion blending. Extensive experiments and cross-validations demonstrate the effectiveness of our approach and dataset for generating convincing two-hand motions from the hybrid body-and-textual conditions. Our dataset and code will be disseminated to the community for future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_07937 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | BOTH2Hands: Inferring 3D Hands from Both Text Prompts and Body Dynamics Zhang, Wenqian Huang, Molin Zhou, Yuxuan Zhang, Juze Yu, Jingyi Wang, Jingya Xu, Lan Computer Vision and Pattern Recognition The recently emerging text-to-motion advances have spired numerous attempts for convenient and interactive human motion generation. Yet, existing methods are largely limited to generating body motions only without considering the rich two-hand motions, let alone handling various conditions like body dynamics or texts. To break the data bottleneck, we propose BOTH57M, a novel multi-modal dataset for two-hand motion generation. Our dataset includes accurate motion tracking for the human body and hands and provides pair-wised finger-level hand annotations and body descriptions. We further provide a strong baseline method, BOTH2Hands, for the novel task: generating vivid two-hand motions from both implicit body dynamics and explicit text prompts. We first warm up two parallel body-to-hand and text-to-hand diffusion models and then utilize the cross-attention transformer for motion blending. Extensive experiments and cross-validations demonstrate the effectiveness of our approach and dataset for generating convincing two-hand motions from the hybrid body-and-textual conditions. Our dataset and code will be disseminated to the community for future research. |
| title | BOTH2Hands: Inferring 3D Hands from Both Text Prompts and Body Dynamics |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2312.07937 |