SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video Diffusion Transformers
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910979670933504 |
|---|---|
| author | Fei, Zhengcong Jiang, Hao Qiu, Di Gu, Baoxuan Zhang, Youqiang Wang, Jiahua Bai, Jialin Li, Debang Fan, Mingyuan Chen, Guibin Zhou, Yahui |
| author_facet | Fei, Zhengcong Jiang, Hao Qiu, Di Gu, Baoxuan Zhang, Youqiang Wang, Jiahua Bai, Jialin Li, Debang Fan, Mingyuan Chen, Guibin Zhou, Yahui |
| contents | The generation and editing of audio-conditioned talking portraits guided by multimodal inputs, including text, images, and videos, remains under explored. In this paper, we present SkyReels-Audio, a unified framework for synthesizing high-fidelity and temporally coherent talking portrait videos. Built upon pretrained video diffusion transformers, our framework supports infinite-length generation and editing, while enabling diverse and controllable conditioning through multimodal inputs. We employ a hybrid curriculum learning strategy to progressively align audio with facial motion, enabling fine-grained multimodal control over long video sequences. To enhance local facial coherence, we introduce a facial mask loss and an audio-guided classifier-free guidance mechanism. A sliding-window denoising approach further fuses latent representations across temporal segments, ensuring visual fidelity and temporal consistency across extended durations and diverse identities. More importantly, we construct a dedicated data pipeline for curating high-quality triplets consisting of synchronized audio, video, and textual descriptions. Comprehensive benchmark evaluations show that SkyReels-Audio achieves superior performance in lip-sync accuracy, identity consistency, and realistic facial dynamics, particularly under complex and challenging conditions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_00830 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video Diffusion Transformers Fei, Zhengcong Jiang, Hao Qiu, Di Gu, Baoxuan Zhang, Youqiang Wang, Jiahua Bai, Jialin Li, Debang Fan, Mingyuan Chen, Guibin Zhou, Yahui Computer Vision and Pattern Recognition The generation and editing of audio-conditioned talking portraits guided by multimodal inputs, including text, images, and videos, remains under explored. In this paper, we present SkyReels-Audio, a unified framework for synthesizing high-fidelity and temporally coherent talking portrait videos. Built upon pretrained video diffusion transformers, our framework supports infinite-length generation and editing, while enabling diverse and controllable conditioning through multimodal inputs. We employ a hybrid curriculum learning strategy to progressively align audio with facial motion, enabling fine-grained multimodal control over long video sequences. To enhance local facial coherence, we introduce a facial mask loss and an audio-guided classifier-free guidance mechanism. A sliding-window denoising approach further fuses latent representations across temporal segments, ensuring visual fidelity and temporal consistency across extended durations and diverse identities. More importantly, we construct a dedicated data pipeline for curating high-quality triplets consisting of synchronized audio, video, and textual descriptions. Comprehensive benchmark evaluations show that SkyReels-Audio achieves superior performance in lip-sync accuracy, identity consistency, and realistic facial dynamics, particularly under complex and challenging conditions. |
| title | SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video Diffusion Transformers |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2506.00830 |