MirrorMe: Towards Realtime and High Fidelity Audio-Driven Halfbody Animation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908425482403840 |
|---|---|
| author | Meng, Dechao Xiao, Steven Zhang, Xindi Wang, Guangyuan Zhang, Peng Wang, Qi Zhang, Bang Bo, Liefeng |
| author_facet | Meng, Dechao Xiao, Steven Zhang, Xindi Wang, Guangyuan Zhang, Peng Wang, Qi Zhang, Bang Bo, Liefeng |
| contents | Audio-driven portrait animation, which synthesizes realistic videos from reference images using audio signals, faces significant challenges in real-time generation of high-fidelity, temporally coherent animations. While recent diffusion-based methods improve generation quality by integrating audio into denoising processes, their reliance on frame-by-frame UNet architectures introduces prohibitive latency and struggles with temporal consistency. This paper introduces MirrorMe, a real-time, controllable framework built on the LTX video model, a diffusion transformer that compresses video spatially and temporally for efficient latent space denoising. To address LTX's trade-offs between compression and semantic fidelity, we propose three innovations: 1. A reference identity injection mechanism via VAE-encoded image concatenation and self-attention, ensuring identity consistency; 2. A causal audio encoder and adapter tailored to LTX's temporal structure, enabling precise audio-expression synchronization; and 3. A progressive training strategy combining close-up facial training, half-body synthesis with facial masking, and hand pose integration for enhanced gesture control. Extensive experiments on the EMTD Benchmark demonstrate MirrorMe's state-of-the-art performance in fidelity, lip-sync accuracy, and temporal stability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_22065 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MirrorMe: Towards Realtime and High Fidelity Audio-Driven Halfbody Animation Meng, Dechao Xiao, Steven Zhang, Xindi Wang, Guangyuan Zhang, Peng Wang, Qi Zhang, Bang Bo, Liefeng Computer Vision and Pattern Recognition Audio-driven portrait animation, which synthesizes realistic videos from reference images using audio signals, faces significant challenges in real-time generation of high-fidelity, temporally coherent animations. While recent diffusion-based methods improve generation quality by integrating audio into denoising processes, their reliance on frame-by-frame UNet architectures introduces prohibitive latency and struggles with temporal consistency. This paper introduces MirrorMe, a real-time, controllable framework built on the LTX video model, a diffusion transformer that compresses video spatially and temporally for efficient latent space denoising. To address LTX's trade-offs between compression and semantic fidelity, we propose three innovations: 1. A reference identity injection mechanism via VAE-encoded image concatenation and self-attention, ensuring identity consistency; 2. A causal audio encoder and adapter tailored to LTX's temporal structure, enabling precise audio-expression synchronization; and 3. A progressive training strategy combining close-up facial training, half-body synthesis with facial masking, and hand pose integration for enhanced gesture control. Extensive experiments on the EMTD Benchmark demonstrate MirrorMe's state-of-the-art performance in fidelity, lip-sync accuracy, and temporal stability. |
| title | MirrorMe: Towards Realtime and High Fidelity Audio-Driven Halfbody Animation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2506.22065 |