MagicPose: Realistic Human Poses and Facial Expressions Retargeting with Identity-aware Diffusion
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909188920180736 |
|---|---|
| author | Chang, Di Shi, Yichun Gao, Quankai Fu, Jessica Xu, Hongyi Song, Guoxian Yan, Qing Zhu, Yizhe Yang, Xiao Soleymani, Mohammad |
| author_facet | Chang, Di Shi, Yichun Gao, Quankai Fu, Jessica Xu, Hongyi Song, Guoxian Yan, Qing Zhu, Yizhe Yang, Xiao Soleymani, Mohammad |
| contents | In this work, we propose MagicPose, a diffusion-based model for 2D human pose and facial expression retargeting. Specifically, given a reference image, we aim to generate a person's new images by controlling the poses and facial expressions while keeping the identity unchanged. To this end, we propose a two-stage training strategy to disentangle human motions and appearance (e.g., facial expressions, skin tone and dressing), consisting of (1) the pre-training of an appearance-control block and (2) learning appearance-disentangled pose control. Our novel design enables robust appearance control over generated human images, including body, facial attributes, and even background. By leveraging the prior knowledge of image diffusion models, MagicPose generalizes well to unseen human identities and complex poses without the need for additional fine-tuning. Moreover, the proposed model is easy to use and can be considered as a plug-in module/extension to Stable Diffusion. The code is available at: https://github.com/Boese0601/MagicDance |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_12052 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | MagicPose: Realistic Human Poses and Facial Expressions Retargeting with Identity-aware Diffusion Chang, Di Shi, Yichun Gao, Quankai Fu, Jessica Xu, Hongyi Song, Guoxian Yan, Qing Zhu, Yizhe Yang, Xiao Soleymani, Mohammad Computer Vision and Pattern Recognition In this work, we propose MagicPose, a diffusion-based model for 2D human pose and facial expression retargeting. Specifically, given a reference image, we aim to generate a person's new images by controlling the poses and facial expressions while keeping the identity unchanged. To this end, we propose a two-stage training strategy to disentangle human motions and appearance (e.g., facial expressions, skin tone and dressing), consisting of (1) the pre-training of an appearance-control block and (2) learning appearance-disentangled pose control. Our novel design enables robust appearance control over generated human images, including body, facial attributes, and even background. By leveraging the prior knowledge of image diffusion models, MagicPose generalizes well to unseen human identities and complex poses without the need for additional fine-tuning. Moreover, the proposed model is easy to use and can be considered as a plug-in module/extension to Stable Diffusion. The code is available at: https://github.com/Boese0601/MagicDance |
| title | MagicPose: Realistic Human Poses and Facial Expressions Retargeting with Identity-aware Diffusion |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2311.12052 |