SayAnything: Audio-Driven Lip Synchronization with Conditional Video Diffusion
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910831377121280 |
|---|---|
| author | Ma, Junxian Wang, Shiwen Yang, Jian Hu, Junyi Liang, Jian Lin, Guosheng chen, Jingbo Li, Kai Meng, Yu |
| author_facet | Ma, Junxian Wang, Shiwen Yang, Jian Hu, Junyi Liang, Jian Lin, Guosheng chen, Jingbo Li, Kai Meng, Yu |
| contents | Recent advances in diffusion models have led to significant progress in audio-driven lip synchronization. However, existing methods typically rely on constrained audio-visual alignment priors or multi-stage learning of intermediate representations to force lip motion synthesis. This leads to complex training pipelines and limited motion naturalness. In this paper, we present SayAnything, a conditional video diffusion framework that directly synthesizes lip movements from audio input while preserving speaker identity. Specifically, we propose three specialized modules including identity preservation module, audio guidance module, and editing control module. Our novel design effectively balances different condition signals in the latent space, enabling precise control over appearance, motion, and region-specific generation without requiring additional supervision signals or intermediate representations. Extensive experiments demonstrate that SayAnything generates highly realistic videos with improved lip-teeth coherence, enabling unseen characters to say anything, while effectively generalizing to animated characters. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_11515 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SayAnything: Audio-Driven Lip Synchronization with Conditional Video Diffusion Ma, Junxian Wang, Shiwen Yang, Jian Hu, Junyi Liang, Jian Lin, Guosheng chen, Jingbo Li, Kai Meng, Yu Computer Vision and Pattern Recognition Recent advances in diffusion models have led to significant progress in audio-driven lip synchronization. However, existing methods typically rely on constrained audio-visual alignment priors or multi-stage learning of intermediate representations to force lip motion synthesis. This leads to complex training pipelines and limited motion naturalness. In this paper, we present SayAnything, a conditional video diffusion framework that directly synthesizes lip movements from audio input while preserving speaker identity. Specifically, we propose three specialized modules including identity preservation module, audio guidance module, and editing control module. Our novel design effectively balances different condition signals in the latent space, enabling precise control over appearance, motion, and region-specific generation without requiring additional supervision signals or intermediate representations. Extensive experiments demonstrate that SayAnything generates highly realistic videos with improved lip-teeth coherence, enabling unseen characters to say anything, while effectively generalizing to animated characters. |
| title | SayAnything: Audio-Driven Lip Synchronization with Conditional Video Diffusion |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2502.11515 |