Joint Learning of Depth and Appearance for Portrait Image Animation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910785584758784 |
|---|---|
| author | Ji, Xinya Zoss, Gaspard Chandran, Prashanth Yang, Lingchen Cao, Xun Solenthaler, Barbara Bradley, Derek |
| author_facet | Ji, Xinya Zoss, Gaspard Chandran, Prashanth Yang, Lingchen Cao, Xun Solenthaler, Barbara Bradley, Derek |
| contents | 2D portrait animation has experienced significant advancements in recent years. Much research has utilized the prior knowledge embedded in large generative diffusion models to enhance high-quality image manipulation. However, most methods only focus on generating RGB images as output, and the co-generation of consistent visual plus 3D output remains largely under-explored. In our work, we propose to jointly learn the visual appearance and depth simultaneously in a diffusion-based portrait image generator. Our method embraces the end-to-end diffusion paradigm and introduces a new architecture suitable for learning this conditional joint distribution, consisting of a reference network and a channel-expanded diffusion backbone. Once trained, our framework can be efficiently adapted to various downstream applications, such as facial depth-to-image and image-to-depth generation, portrait relighting, and audio-driven talking head animation with consistent 3D output. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_08649 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Joint Learning of Depth and Appearance for Portrait Image Animation Ji, Xinya Zoss, Gaspard Chandran, Prashanth Yang, Lingchen Cao, Xun Solenthaler, Barbara Bradley, Derek Computer Vision and Pattern Recognition Machine Learning 2D portrait animation has experienced significant advancements in recent years. Much research has utilized the prior knowledge embedded in large generative diffusion models to enhance high-quality image manipulation. However, most methods only focus on generating RGB images as output, and the co-generation of consistent visual plus 3D output remains largely under-explored. In our work, we propose to jointly learn the visual appearance and depth simultaneously in a diffusion-based portrait image generator. Our method embraces the end-to-end diffusion paradigm and introduces a new architecture suitable for learning this conditional joint distribution, consisting of a reference network and a channel-expanded diffusion backbone. Once trained, our framework can be efficiently adapted to various downstream applications, such as facial depth-to-image and image-to-depth generation, portrait relighting, and audio-driven talking head animation with consistent 3D output. |
| title | Joint Learning of Depth and Appearance for Portrait Image Animation |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2501.08649 |