Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2412.02419 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917855604244480 |
|---|---|
| author | Shi, Mingyi Qin, Dafei Ho, Leo Liao, Zhouyingcheng Huang, Yinghao Yamagishi, Junichi Komura, Taku |
| author_facet | Shi, Mingyi Qin, Dafei Ho, Leo Liao, Zhouyingcheng Huang, Yinghao Yamagishi, Junichi Komura, Taku |
| contents | Conversational scenarios are very common in real-world settings, yet existing co-speech motion synthesis approaches often fall short in these contexts, where one person's audio and gestures will influence the other's responses. Additionally, most existing methods rely on offline sequence-to-sequence frameworks, which are unsuitable for online applications. In this work, we introduce an audio-driven, auto-regressive system designed to synthesize dynamic movements for two characters during a conversation. At the core of our approach is a diffusion-based full-body motion synthesis model, which is conditioned on the past states of both characters, speech audio, and a task-oriented motion trajectory input, allowing for flexible spatial control. To enhance the model's ability to learn diverse interactions, we have enriched existing two-person conversational motion datasets with more dynamic and interactive motions. We evaluate our system through multiple experiments to show it outperforms across a variety of tasks, including single and two-person co-speech motion generation, as well as interactive motion generation. To the best of our knowledge, this is the first system capable of generating interactive full-body motions for two characters from speech in an online manner. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_02419 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | It Takes Two: Real-time Co-Speech Two-person's Interaction Generation via Reactive Auto-regressive Diffusion Model Shi, Mingyi Qin, Dafei Ho, Leo Liao, Zhouyingcheng Huang, Yinghao Yamagishi, Junichi Komura, Taku Sound Computer Vision and Pattern Recognition Graphics Multimedia Audio and Speech Processing Conversational scenarios are very common in real-world settings, yet existing co-speech motion synthesis approaches often fall short in these contexts, where one person's audio and gestures will influence the other's responses. Additionally, most existing methods rely on offline sequence-to-sequence frameworks, which are unsuitable for online applications. In this work, we introduce an audio-driven, auto-regressive system designed to synthesize dynamic movements for two characters during a conversation. At the core of our approach is a diffusion-based full-body motion synthesis model, which is conditioned on the past states of both characters, speech audio, and a task-oriented motion trajectory input, allowing for flexible spatial control. To enhance the model's ability to learn diverse interactions, we have enriched existing two-person conversational motion datasets with more dynamic and interactive motions. We evaluate our system through multiple experiments to show it outperforms across a variety of tasks, including single and two-person co-speech motion generation, as well as interactive motion generation. To the best of our knowledge, this is the first system capable of generating interactive full-body motions for two characters from speech in an online manner. |
| title | It Takes Two: Real-time Co-Speech Two-person's Interaction Generation via Reactive Auto-regressive Diffusion Model |
| topic | Sound Computer Vision and Pattern Recognition Graphics Multimedia Audio and Speech Processing |
| url | https://arxiv.org/abs/2412.02419 |