UniAnimate-DiT: Human Image Animation with Large-Scale Video Diffusion Transformer
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917355735482368 |
|---|---|
| author | Wang, Xiang Zhang, Shiwei Tang, Longxiang Zhang, Yingya Gao, Changxin Wang, Yuehuan Sang, Nong |
| author_facet | Wang, Xiang Zhang, Shiwei Tang, Longxiang Zhang, Yingya Gao, Changxin Wang, Yuehuan Sang, Nong |
| contents | This report presents UniAnimate-DiT, an advanced project that leverages the cutting-edge and powerful capabilities of the open-source Wan2.1 model for consistent human image animation. Specifically, to preserve the robust generative capabilities of the original Wan2.1 model, we implement Low-Rank Adaptation (LoRA) technique to fine-tune a minimal set of parameters, significantly reducing training memory overhead. A lightweight pose encoder consisting of multiple stacked 3D convolutional layers is designed to encode motion information of driving poses. Furthermore, we adopt a simple concatenation operation to integrate the reference appearance into the model and incorporate the pose information of the reference image for enhanced pose alignment. Experimental results show that our approach achieves visually appearing and temporally consistent high-fidelity animations. Trained on 480p (832x480) videos, UniAnimate-DiT demonstrates strong generalization capabilities to seamlessly upscale to 720P (1280x720) during inference. The training and inference code is publicly available at https://github.com/ali-vilab/UniAnimate-DiT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_11289 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | UniAnimate-DiT: Human Image Animation with Large-Scale Video Diffusion Transformer Wang, Xiang Zhang, Shiwei Tang, Longxiang Zhang, Yingya Gao, Changxin Wang, Yuehuan Sang, Nong Computer Vision and Pattern Recognition Multimedia This report presents UniAnimate-DiT, an advanced project that leverages the cutting-edge and powerful capabilities of the open-source Wan2.1 model for consistent human image animation. Specifically, to preserve the robust generative capabilities of the original Wan2.1 model, we implement Low-Rank Adaptation (LoRA) technique to fine-tune a minimal set of parameters, significantly reducing training memory overhead. A lightweight pose encoder consisting of multiple stacked 3D convolutional layers is designed to encode motion information of driving poses. Furthermore, we adopt a simple concatenation operation to integrate the reference appearance into the model and incorporate the pose information of the reference image for enhanced pose alignment. Experimental results show that our approach achieves visually appearing and temporally consistent high-fidelity animations. Trained on 480p (832x480) videos, UniAnimate-DiT demonstrates strong generalization capabilities to seamlessly upscale to 720P (1280x720) during inference. The training and inference code is publicly available at https://github.com/ali-vilab/UniAnimate-DiT. |
| title | UniAnimate-DiT: Human Image Animation with Large-Scale Video Diffusion Transformer |
| topic | Computer Vision and Pattern Recognition Multimedia |
| url | https://arxiv.org/abs/2504.11289 |