Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.10851 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910659682238464 |
|---|---|
| author | Pang, Haozhou Ding, Tianwei He, Lanshan Tao, Ming Zhang, Lu Gan, Qi |
| author_facet | Pang, Haozhou Ding, Tianwei He, Lanshan Tao, Ming Zhang, Lu Gan, Qi |
| contents | In this work, we present LLM Gesticulator, an LLM-based audio-driven co-speech gesture generation framework that synthesizes full-body animations that are rhythmically aligned with the input audio while exhibiting natural movements and editability. Compared to previous work, our model demonstrates substantial scalability. As the size of the backbone LLM model increases, our framework shows proportional improvements in evaluation metrics (a.k.a. scaling law). Our method also exhibits strong controllability where the content, style of the generated gestures can be controlled by text prompt. To the best of our knowledge, LLM gesticulator is the first work that use LLM on the co-speech generation task. Evaluation with existing objective metrics and user studies indicate that our framework outperforms prior works. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_10851 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LLM Gesticulator: Leveraging Large Language Models for Scalable and Controllable Co-Speech Gesture Synthesis Pang, Haozhou Ding, Tianwei He, Lanshan Tao, Ming Zhang, Lu Gan, Qi Graphics Artificial Intelligence Computation and Language Machine Learning Sound Audio and Speech Processing In this work, we present LLM Gesticulator, an LLM-based audio-driven co-speech gesture generation framework that synthesizes full-body animations that are rhythmically aligned with the input audio while exhibiting natural movements and editability. Compared to previous work, our model demonstrates substantial scalability. As the size of the backbone LLM model increases, our framework shows proportional improvements in evaluation metrics (a.k.a. scaling law). Our method also exhibits strong controllability where the content, style of the generated gestures can be controlled by text prompt. To the best of our knowledge, LLM gesticulator is the first work that use LLM on the co-speech generation task. Evaluation with existing objective metrics and user studies indicate that our framework outperforms prior works. |
| title | LLM Gesticulator: Leveraging Large Language Models for Scalable and Controllable Co-Speech Gesture Synthesis |
| topic | Graphics Artificial Intelligence Computation and Language Machine Learning Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2410.10851 |