Coherent 3D Portrait Video Reconstruction via Triplane Fusion
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929333352792064 |
|---|---|
| author | Wang, Shengze Li, Xueting Liu, Chao Chan, Matthew Stengel, Michael Spjut, Josef Fuchs, Henry De Mello, Shalini Nagano, Koki |
| author_facet | Wang, Shengze Li, Xueting Liu, Chao Chan, Matthew Stengel, Michael Spjut, Josef Fuchs, Henry De Mello, Shalini Nagano, Koki |
| contents | Recent breakthroughs in single-image 3D portrait reconstruction have enabled telepresence systems to stream 3D portrait videos from a single camera in real-time, potentially democratizing telepresence. However, per-frame 3D reconstruction exhibits temporal inconsistency and forgets the user's appearance. On the other hand, self-reenactment methods can render coherent 3D portraits by driving a personalized 3D prior, but fail to faithfully reconstruct the user's per-frame appearance (e.g., facial expressions and lighting). In this work, we recognize the need to maintain both coherent identity and dynamic per-frame appearance to enable the best possible realism. To this end, we propose a new fusion-based method that fuses a personalized 3D subject prior with per-frame information, producing temporally stable 3D videos with faithful reconstruction of the user's per-frame appearances. Trained only using synthetic data produced by an expression-conditioned 3D GAN, our encoder-based method achieves both state-of-the-art 3D reconstruction accuracy and temporal consistency on in-studio and in-the-wild datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_00794 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Coherent 3D Portrait Video Reconstruction via Triplane Fusion Wang, Shengze Li, Xueting Liu, Chao Chan, Matthew Stengel, Michael Spjut, Josef Fuchs, Henry De Mello, Shalini Nagano, Koki Computer Vision and Pattern Recognition Recent breakthroughs in single-image 3D portrait reconstruction have enabled telepresence systems to stream 3D portrait videos from a single camera in real-time, potentially democratizing telepresence. However, per-frame 3D reconstruction exhibits temporal inconsistency and forgets the user's appearance. On the other hand, self-reenactment methods can render coherent 3D portraits by driving a personalized 3D prior, but fail to faithfully reconstruct the user's per-frame appearance (e.g., facial expressions and lighting). In this work, we recognize the need to maintain both coherent identity and dynamic per-frame appearance to enable the best possible realism. To this end, we propose a new fusion-based method that fuses a personalized 3D subject prior with per-frame information, producing temporally stable 3D videos with faithful reconstruction of the user's per-frame appearances. Trained only using synthetic data produced by an expression-conditioned 3D GAN, our encoder-based method achieves both state-of-the-art 3D reconstruction accuracy and temporal consistency on in-studio and in-the-wild datasets. |
| title | Coherent 3D Portrait Video Reconstruction via Triplane Fusion |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2405.00794 |