Real-time One-Step Diffusion-based Expressive Portrait Videos Generation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912160298303488 |
|---|---|
| author | Guo, Hanzhong Yi, Hongwei Zhou, Daquan Bergman, Alexander William Lingelbach, Michael Yu, Yizhou |
| author_facet | Guo, Hanzhong Yi, Hongwei Zhou, Daquan Bergman, Alexander William Lingelbach, Michael Yu, Yizhou |
| contents | Latent diffusion models have made great strides in generating expressive portrait videos with accurate lip-sync and natural motion from a single reference image and audio input. However, these models are far from real-time, often requiring many sampling steps that take minutes to generate even one second of video-significantly limiting practical use. We introduce OSA-LCM (One-Step Avatar Latent Consistency Model), paving the way for real-time diffusion-based avatars. Our method achieves comparable video quality to existing methods but requires only one sampling step, making it more than 10x faster. To accomplish this, we propose a novel avatar discriminator design that guides lip-audio consistency and motion expressiveness to enhance video quality in limited sampling steps. Additionally, we employ a second-stage training architecture using an editing fine-tuned method (EFT), transforming video generation into an editing task during training to effectively address the temporal gap challenge in single-step generation. Experiments demonstrate that OSA-LCM outperforms existing open-source portrait video generation models while operating more efficiently with a single sampling step. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_13479 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Real-time One-Step Diffusion-based Expressive Portrait Videos Generation Guo, Hanzhong Yi, Hongwei Zhou, Daquan Bergman, Alexander William Lingelbach, Michael Yu, Yizhou Computer Vision and Pattern Recognition Latent diffusion models have made great strides in generating expressive portrait videos with accurate lip-sync and natural motion from a single reference image and audio input. However, these models are far from real-time, often requiring many sampling steps that take minutes to generate even one second of video-significantly limiting practical use. We introduce OSA-LCM (One-Step Avatar Latent Consistency Model), paving the way for real-time diffusion-based avatars. Our method achieves comparable video quality to existing methods but requires only one sampling step, making it more than 10x faster. To accomplish this, we propose a novel avatar discriminator design that guides lip-audio consistency and motion expressiveness to enhance video quality in limited sampling steps. Additionally, we employ a second-stage training architecture using an editing fine-tuned method (EFT), transforming video generation into an editing task during training to effectively address the temporal gap challenge in single-step generation. Experiments demonstrate that OSA-LCM outperforms existing open-source portrait video generation models while operating more efficiently with a single sampling step. |
| title | Real-time One-Step Diffusion-based Expressive Portrait Videos Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.13479 |