MaTe3D: Mask-guided Text-based 3D-aware Portrait Editing
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929410029912064 |
|---|---|
| author | Zhou, Kangneng Gao, Daiheng Wang, Xuan Zhang, Jie Zhang, Peng Sun, Xusen Zhang, Longhao Yang, Shiqi Zhang, Bang Bo, Liefeng Wang, Yaxing Cheng, Ming-Ming |
| author_facet | Zhou, Kangneng Gao, Daiheng Wang, Xuan Zhang, Jie Zhang, Peng Sun, Xusen Zhang, Longhao Yang, Shiqi Zhang, Bang Bo, Liefeng Wang, Yaxing Cheng, Ming-Ming |
| contents | 3D-aware portrait editing has a wide range of applications in multiple fields. However, current approaches are limited due that they can only perform mask-guided or text-based editing. Even by fusing the two procedures into a model, the editing quality and stability cannot be ensured. To address this limitation, we propose \textbf{MaTe3D}: mask-guided text-based 3D-aware portrait editing. In this framework, first, we introduce a new SDF-based 3D generator which learns local and global representations with proposed SDF and density consistency losses. This enhances masked-based editing in local areas; second, we present a novel distillation strategy: Conditional Distillation on Geometry and Texture (CDGT). Compared to exiting distillation strategies, it mitigates visual ambiguity and avoids mismatch between texture and geometry, thereby producing stable texture and convincing geometry while editing. Additionally, we create the CatMask-HQ dataset, a large-scale high-resolution cat face annotation for exploration of model generalization and expansion. We perform expensive experiments on both the FFHQ and CatMask-HQ datasets to demonstrate the editing quality and stability of the proposed method. Our method faithfully generates a 3D-aware edited face image based on a modified mask and a text prompt. Our code and models will be publicly released. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_06947 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | MaTe3D: Mask-guided Text-based 3D-aware Portrait Editing Zhou, Kangneng Gao, Daiheng Wang, Xuan Zhang, Jie Zhang, Peng Sun, Xusen Zhang, Longhao Yang, Shiqi Zhang, Bang Bo, Liefeng Wang, Yaxing Cheng, Ming-Ming Computer Vision and Pattern Recognition 3D-aware portrait editing has a wide range of applications in multiple fields. However, current approaches are limited due that they can only perform mask-guided or text-based editing. Even by fusing the two procedures into a model, the editing quality and stability cannot be ensured. To address this limitation, we propose \textbf{MaTe3D}: mask-guided text-based 3D-aware portrait editing. In this framework, first, we introduce a new SDF-based 3D generator which learns local and global representations with proposed SDF and density consistency losses. This enhances masked-based editing in local areas; second, we present a novel distillation strategy: Conditional Distillation on Geometry and Texture (CDGT). Compared to exiting distillation strategies, it mitigates visual ambiguity and avoids mismatch between texture and geometry, thereby producing stable texture and convincing geometry while editing. Additionally, we create the CatMask-HQ dataset, a large-scale high-resolution cat face annotation for exploration of model generalization and expansion. We perform expensive experiments on both the FFHQ and CatMask-HQ datasets to demonstrate the editing quality and stability of the proposed method. Our method faithfully generates a 3D-aware edited face image based on a modified mask and a text prompt. Our code and models will be publicly released. |
| title | MaTe3D: Mask-guided Text-based 3D-aware Portrait Editing |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2312.06947 |