In-N-Out: Faithful 3D GAN Inversion with Volumetric Decomposition for Face Editing
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914751716524032 |
|---|---|
| author | Xu, Yiran Shu, Zhixin Smith, Cameron Oh, Seoung Wug Huang, Jia-Bin |
| author_facet | Xu, Yiran Shu, Zhixin Smith, Cameron Oh, Seoung Wug Huang, Jia-Bin |
| contents | 3D-aware GANs offer new capabilities for view synthesis while preserving the editing functionalities of their 2D counterparts. GAN inversion is a crucial step that seeks the latent code to reconstruct input images or videos, subsequently enabling diverse editing tasks through manipulation of this latent code. However, a model pre-trained on a particular dataset (e.g., FFHQ) often has difficulty reconstructing images with out-of-distribution (OOD) objects such as faces with heavy make-up or occluding objects. We address this issue by explicitly modeling OOD objects from the input in 3D-aware GANs. Our core idea is to represent the image using two individual neural radiance fields: one for the in-distribution content and the other for the out-of-distribution object. The final reconstruction is achieved by optimizing the composition of these two radiance fields with carefully designed regularization. We demonstrate that our explicit decomposition alleviates the inherent trade-off between reconstruction fidelity and editability. We evaluate reconstruction accuracy and editability of our method on challenging real face images and videos and showcase favorable results against other baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2302_04871 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | In-N-Out: Faithful 3D GAN Inversion with Volumetric Decomposition for Face Editing Xu, Yiran Shu, Zhixin Smith, Cameron Oh, Seoung Wug Huang, Jia-Bin Computer Vision and Pattern Recognition 3D-aware GANs offer new capabilities for view synthesis while preserving the editing functionalities of their 2D counterparts. GAN inversion is a crucial step that seeks the latent code to reconstruct input images or videos, subsequently enabling diverse editing tasks through manipulation of this latent code. However, a model pre-trained on a particular dataset (e.g., FFHQ) often has difficulty reconstructing images with out-of-distribution (OOD) objects such as faces with heavy make-up or occluding objects. We address this issue by explicitly modeling OOD objects from the input in 3D-aware GANs. Our core idea is to represent the image using two individual neural radiance fields: one for the in-distribution content and the other for the out-of-distribution object. The final reconstruction is achieved by optimizing the composition of these two radiance fields with carefully designed regularization. We demonstrate that our explicit decomposition alleviates the inherent trade-off between reconstruction fidelity and editability. We evaluate reconstruction accuracy and editability of our method on challenging real face images and videos and showcase favorable results against other baselines. |
| title | In-N-Out: Faithful 3D GAN Inversion with Volumetric Decomposition for Face Editing |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2302.04871 |