Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913864464990208 |
|---|---|
| author | Kong, Zhe Gao, Feng Zhang, Yong Kang, Zhuoliang Wei, Xiaoming Cai, Xunliang Chen, Guanying Luo, Wenhan |
| author_facet | Kong, Zhe Gao, Feng Zhang, Yong Kang, Zhuoliang Wei, Xiaoming Cai, Xunliang Chen, Guanying Luo, Wenhan |
| contents | Audio-driven human animation methods, such as talking head and talking body generation, have made remarkable progress in generating synchronized facial movements and appealing visual quality videos. However, existing methods primarily focus on single human animation and struggle with multi-stream audio inputs, facing incorrect binding problems between audio and persons. Additionally, they exhibit limitations in instruction-following capabilities. To solve this problem, in this paper, we propose a novel task: Multi-Person Conversational Video Generation, and introduce a new framework, MultiTalk, to address the challenges during multi-person generation. Specifically, for audio injection, we investigate several schemes and propose the Label Rotary Position Embedding (L-RoPE) method to resolve the audio and person binding problem. Furthermore, during training, we observe that partial parameter training and multi-task training are crucial for preserving the instruction-following ability of the base model. MultiTalk achieves superior performance compared to other methods on several datasets, including talking head, talking body, and multi-person datasets, demonstrating the powerful generation capabilities of our approach. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_22647 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation Kong, Zhe Gao, Feng Zhang, Yong Kang, Zhuoliang Wei, Xiaoming Cai, Xunliang Chen, Guanying Luo, Wenhan Computer Vision and Pattern Recognition Audio-driven human animation methods, such as talking head and talking body generation, have made remarkable progress in generating synchronized facial movements and appealing visual quality videos. However, existing methods primarily focus on single human animation and struggle with multi-stream audio inputs, facing incorrect binding problems between audio and persons. Additionally, they exhibit limitations in instruction-following capabilities. To solve this problem, in this paper, we propose a novel task: Multi-Person Conversational Video Generation, and introduce a new framework, MultiTalk, to address the challenges during multi-person generation. Specifically, for audio injection, we investigate several schemes and propose the Label Rotary Position Embedding (L-RoPE) method to resolve the audio and person binding problem. Furthermore, during training, we observe that partial parameter training and multi-task training are crucial for preserving the instruction-following ability of the base model. MultiTalk achieves superior performance compared to other methods on several datasets, including talking head, talking body, and multi-person datasets, demonstrating the powerful generation capabilities of our approach. |
| title | Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2505.22647 |