Soul: Breathe Life into Digital Human for High-fidelity Long-term Multimodal Animation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911320060723200 |
|---|---|
| author | Zhang, Jiangning Zhu, Junwei Gan, Zhenye Luo, Donghao Lin, Chuming Xu, Feifan Peng, Xu Hu, Jianlong Liu, Yuansen Hong, Yijia Cao, Weijian Feng, Han Chen, Xu Fu, Chencan He, Keke Hu, Xiaobin Wang, Chengjie |
| author_facet | Zhang, Jiangning Zhu, Junwei Gan, Zhenye Luo, Donghao Lin, Chuming Xu, Feifan Peng, Xu Hu, Jianlong Liu, Yuansen Hong, Yijia Cao, Weijian Feng, Han Chen, Xu Fu, Chencan He, Keke Hu, Xiaobin Wang, Chengjie |
| contents | We propose a multimodal-driven framework for high-fidelity long-term digital human animation termed $\textbf{Soul}$, which generates semantically coherent videos from a single-frame portrait image, text prompts, and audio, achieving precise lip synchronization, vivid facial expressions, and robust identity preservation. We construct Soul-1M, containing 1 million finely annotated samples with a precise automated annotation pipeline (covering portrait, upper-body, full-body, and multi-person scenes) to mitigate data scarcity, and we carefully curate Soul-Bench for comprehensive and fair evaluation of audio-/text-guided animation methods. The model is built on the Wan2.2-5B backbone, integrating audio-injection layers and multiple training strategies together with threshold-aware codebook replacement to ensure long-term generation consistency. Meanwhile, step/CFG distillation and a lightweight VAE are used to optimize inference efficiency, achieving an 11.4$\times$ speedup with negligible quality loss. Extensive experiments show that Soul significantly outperforms current leading open-source and commercial models on video quality, video-text alignment, identity preservation, and lip-synchronization accuracy, demonstrating broad applicability in real-world scenarios such as virtual anchors and film production. Project page at https://zhangzjn.github.io/projects/Soul/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_13495 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Soul: Breathe Life into Digital Human for High-fidelity Long-term Multimodal Animation Zhang, Jiangning Zhu, Junwei Gan, Zhenye Luo, Donghao Lin, Chuming Xu, Feifan Peng, Xu Hu, Jianlong Liu, Yuansen Hong, Yijia Cao, Weijian Feng, Han Chen, Xu Fu, Chencan He, Keke Hu, Xiaobin Wang, Chengjie Computer Vision and Pattern Recognition We propose a multimodal-driven framework for high-fidelity long-term digital human animation termed $\textbf{Soul}$, which generates semantically coherent videos from a single-frame portrait image, text prompts, and audio, achieving precise lip synchronization, vivid facial expressions, and robust identity preservation. We construct Soul-1M, containing 1 million finely annotated samples with a precise automated annotation pipeline (covering portrait, upper-body, full-body, and multi-person scenes) to mitigate data scarcity, and we carefully curate Soul-Bench for comprehensive and fair evaluation of audio-/text-guided animation methods. The model is built on the Wan2.2-5B backbone, integrating audio-injection layers and multiple training strategies together with threshold-aware codebook replacement to ensure long-term generation consistency. Meanwhile, step/CFG distillation and a lightweight VAE are used to optimize inference efficiency, achieving an 11.4$\times$ speedup with negligible quality loss. Extensive experiments show that Soul significantly outperforms current leading open-source and commercial models on video quality, video-text alignment, identity preservation, and lip-synchronization accuracy, demonstrating broad applicability in real-world scenarios such as virtual anchors and film production. Project page at https://zhangzjn.github.io/projects/Soul/ |
| title | Soul: Breathe Life into Digital Human for High-fidelity Long-term Multimodal Animation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2512.13495 |