Cafca: High-quality Novel View Synthesis of Expressive Faces from Casual Few-shot Captures
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917792326877184 |
|---|---|
| author | Bühler, Marcel C. Li, Gengyan Wood, Erroll Helminger, Leonhard Chen, Xu Shah, Tanmay Wang, Daoye Garbin, Stephan Orts-Escolano, Sergio Hilliges, Otmar Lagun, Dmitry Riviere, Jérémy Gotardo, Paulo Beeler, Thabo Meka, Abhimitra Sarkar, Kripasindhu |
| author_facet | Bühler, Marcel C. Li, Gengyan Wood, Erroll Helminger, Leonhard Chen, Xu Shah, Tanmay Wang, Daoye Garbin, Stephan Orts-Escolano, Sergio Hilliges, Otmar Lagun, Dmitry Riviere, Jérémy Gotardo, Paulo Beeler, Thabo Meka, Abhimitra Sarkar, Kripasindhu |
| contents | Volumetric modeling and neural radiance field representations have revolutionized 3D face capture and photorealistic novel view synthesis. However, these methods often require hundreds of multi-view input images and are thus inapplicable to cases with less than a handful of inputs. We present a novel volumetric prior on human faces that allows for high-fidelity expressive face modeling from as few as three input views captured in the wild. Our key insight is that an implicit prior trained on synthetic data alone can generalize to extremely challenging real-world identities and expressions and render novel views with fine idiosyncratic details like wrinkles and eyelashes. We leverage a 3D Morphable Face Model to synthesize a large training set, rendering each identity with different expressions, hair, clothing, and other assets. We then train a conditional Neural Radiance Field prior on this synthetic dataset and, at inference time, fine-tune the model on a very sparse set of real images of a single subject. On average, the fine-tuning requires only three inputs to cross the synthetic-to-real domain gap. The resulting personalized 3D model reconstructs strong idiosyncratic facial expressions and outperforms the state-of-the-art in high-quality novel view synthesis of faces from sparse inputs in terms of perceptual and photo-metric quality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_00630 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Cafca: High-quality Novel View Synthesis of Expressive Faces from Casual Few-shot Captures Bühler, Marcel C. Li, Gengyan Wood, Erroll Helminger, Leonhard Chen, Xu Shah, Tanmay Wang, Daoye Garbin, Stephan Orts-Escolano, Sergio Hilliges, Otmar Lagun, Dmitry Riviere, Jérémy Gotardo, Paulo Beeler, Thabo Meka, Abhimitra Sarkar, Kripasindhu Computer Vision and Pattern Recognition Artificial Intelligence Volumetric modeling and neural radiance field representations have revolutionized 3D face capture and photorealistic novel view synthesis. However, these methods often require hundreds of multi-view input images and are thus inapplicable to cases with less than a handful of inputs. We present a novel volumetric prior on human faces that allows for high-fidelity expressive face modeling from as few as three input views captured in the wild. Our key insight is that an implicit prior trained on synthetic data alone can generalize to extremely challenging real-world identities and expressions and render novel views with fine idiosyncratic details like wrinkles and eyelashes. We leverage a 3D Morphable Face Model to synthesize a large training set, rendering each identity with different expressions, hair, clothing, and other assets. We then train a conditional Neural Radiance Field prior on this synthetic dataset and, at inference time, fine-tune the model on a very sparse set of real images of a single subject. On average, the fine-tuning requires only three inputs to cross the synthetic-to-real domain gap. The resulting personalized 3D model reconstructs strong idiosyncratic facial expressions and outperforms the state-of-the-art in high-quality novel view synthesis of faces from sparse inputs in terms of perceptual and photo-metric quality. |
| title | Cafca: High-quality Novel View Synthesis of Expressive Faces from Casual Few-shot Captures |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2410.00630 |