AnyTalker: Scaling Multi-Person Talking Video Generation with Interactivity Refinement
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914174101094400 |
|---|---|
| author | Zhong, Zhizhou Ji, Yicheng Kong, Zhe Liu, Yiying Wang, Jiarui Feng, Jiasun Liu, Lupeng Wang, Xiangyi Li, Yanjia She, Yuqing Qin, Ying Li, Huan Mao, Shuiyang Liu, Wei Luo, Wenhan |
| author_facet | Zhong, Zhizhou Ji, Yicheng Kong, Zhe Liu, Yiying Wang, Jiarui Feng, Jiasun Liu, Lupeng Wang, Xiangyi Li, Yanjia She, Yuqing Qin, Ying Li, Huan Mao, Shuiyang Liu, Wei Luo, Wenhan |
| contents | Recently, multi-person video generation has started to gain prominence. While a few preliminary works have explored audio-driven multi-person talking video generation, they often face challenges due to the high costs of diverse multi-person data collection and the difficulty of driving multiple identities with coherent interactivity. To address these challenges, we propose AnyTalker, a multi-person generation framework that features an extensible multi-stream processing architecture. Specifically, we extend Diffusion Transformer's attention block with a novel identity-aware attention mechanism that iteratively processes identity-audio pairs, allowing arbitrary scaling of drivable identities. Besides, training multi-person generative models demands massive multi-person data. Our proposed training pipeline depends solely on single-person videos to learn multi-person speaking patterns and refines interactivity with only a few real multi-person clips. Furthermore, we contribute a targeted metric and dataset designed to evaluate the naturalness and interactivity of the generated multi-person videos. Extensive experiments demonstrate that AnyTalker achieves remarkable lip synchronization, visual quality, and natural interactivity, striking a favorable balance between data costs and identity scalability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_23475 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AnyTalker: Scaling Multi-Person Talking Video Generation with Interactivity Refinement Zhong, Zhizhou Ji, Yicheng Kong, Zhe Liu, Yiying Wang, Jiarui Feng, Jiasun Liu, Lupeng Wang, Xiangyi Li, Yanjia She, Yuqing Qin, Ying Li, Huan Mao, Shuiyang Liu, Wei Luo, Wenhan Computer Vision and Pattern Recognition Recently, multi-person video generation has started to gain prominence. While a few preliminary works have explored audio-driven multi-person talking video generation, they often face challenges due to the high costs of diverse multi-person data collection and the difficulty of driving multiple identities with coherent interactivity. To address these challenges, we propose AnyTalker, a multi-person generation framework that features an extensible multi-stream processing architecture. Specifically, we extend Diffusion Transformer's attention block with a novel identity-aware attention mechanism that iteratively processes identity-audio pairs, allowing arbitrary scaling of drivable identities. Besides, training multi-person generative models demands massive multi-person data. Our proposed training pipeline depends solely on single-person videos to learn multi-person speaking patterns and refines interactivity with only a few real multi-person clips. Furthermore, we contribute a targeted metric and dataset designed to evaluate the naturalness and interactivity of the generated multi-person videos. Extensive experiments demonstrate that AnyTalker achieves remarkable lip synchronization, visual quality, and natural interactivity, striking a favorable balance between data costs and identity scalability. |
| title | AnyTalker: Scaling Multi-Person Talking Video Generation with Interactivity Refinement |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2511.23475 |