READ: Real-time and Efficient Asynchronous Diffusion for Audio-driven Talking Head Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908655531589632 |
|---|---|
| author | Wang, Haotian Weng, Yuzhe Du, Jun Xu, Haoran Wu, Xiaoyan He, Shan Yin, Bing Liu, Cong Gao, Jianqing Liu, Qingfeng |
| author_facet | Wang, Haotian Weng, Yuzhe Du, Jun Xu, Haoran Wu, Xiaoyan He, Shan Yin, Bing Liu, Cong Gao, Jianqing Liu, Qingfeng |
| contents | The introduction of diffusion models has brought significant advances to the field of audio-driven talking head generation. However, the extremely slow inference speed severely limits the practical implementation of diffusion-based talking head generation models. In this study, we propose READ, a real-time diffusion-transformer-based talking head generation framework. Our approach first learns a spatiotemporal highly compressed video latent space via a temporal VAE, significantly reducing the token count to accelerate generation. To achieve better audio-visual alignment within this compressed latent space, a pre-trained Speech Autoencoder (SpeechAE) is proposed to generate temporally compressed speech latent codes corresponding to the video latent space. These latent representations are then modeled by a carefully designed Audio-to-Video Diffusion Transformer (A2V-DiT) backbone for efficient talking head synthesis. Furthermore, to ensure temporal consistency and accelerated inference in extended generation, we propose a novel asynchronous noise scheduler (ANS) for both the training and inference processes of our framework. The ANS leverages asynchronous add-noise and asynchronous motion-guided generation in the latent space, ensuring consistency in generated video clips. Experimental results demonstrate that READ outperforms state-of-the-art methods by generating competitive talking head videos with significantly reduced runtime, achieving an optimal balance between quality and speed while maintaining robust metric stability in long-time generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_03457 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | READ: Real-time and Efficient Asynchronous Diffusion for Audio-driven Talking Head Generation Wang, Haotian Weng, Yuzhe Du, Jun Xu, Haoran Wu, Xiaoyan He, Shan Yin, Bing Liu, Cong Gao, Jianqing Liu, Qingfeng Graphics Computer Vision and Pattern Recognition Sound Audio and Speech Processing The introduction of diffusion models has brought significant advances to the field of audio-driven talking head generation. However, the extremely slow inference speed severely limits the practical implementation of diffusion-based talking head generation models. In this study, we propose READ, a real-time diffusion-transformer-based talking head generation framework. Our approach first learns a spatiotemporal highly compressed video latent space via a temporal VAE, significantly reducing the token count to accelerate generation. To achieve better audio-visual alignment within this compressed latent space, a pre-trained Speech Autoencoder (SpeechAE) is proposed to generate temporally compressed speech latent codes corresponding to the video latent space. These latent representations are then modeled by a carefully designed Audio-to-Video Diffusion Transformer (A2V-DiT) backbone for efficient talking head synthesis. Furthermore, to ensure temporal consistency and accelerated inference in extended generation, we propose a novel asynchronous noise scheduler (ANS) for both the training and inference processes of our framework. The ANS leverages asynchronous add-noise and asynchronous motion-guided generation in the latent space, ensuring consistency in generated video clips. Experimental results demonstrate that READ outperforms state-of-the-art methods by generating competitive talking head videos with significantly reduced runtime, achieving an optimal balance between quality and speed while maintaining robust metric stability in long-time generation. |
| title | READ: Real-time and Efficient Asynchronous Diffusion for Audio-driven Talking Head Generation |
| topic | Graphics Computer Vision and Pattern Recognition Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2508.03457 |