MCMat: Multiview-Consistent and Physically Accurate PBR Material Generation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913617191895040 |
|---|---|
| author | Zhu, Shenhao Qiu, Lingteng Gu, Xiaodong Zhao, Zhengyi Xu, Chao He, Yuxiao Li, Zhe Han, Xiaoguang Yao, Yao Cao, Xun Zhu, Siyu Yuan, Weihao Dong, Zilong Zhu, Hao |
| author_facet | Zhu, Shenhao Qiu, Lingteng Gu, Xiaodong Zhao, Zhengyi Xu, Chao He, Yuxiao Li, Zhe Han, Xiaoguang Yao, Yao Cao, Xun Zhu, Siyu Yuan, Weihao Dong, Zilong Zhu, Hao |
| contents | Existing 2D methods utilize UNet-based diffusion models to generate multi-view physically-based rendering (PBR) maps but struggle with multi-view inconsistency, while some 3D methods directly generate UV maps, encountering generalization issues due to the limited 3D data. To address these problems, we propose a two-stage approach, including multi-view generation and UV materials refinement. In the generation stage, we adopt a Diffusion Transformer (DiT) model to generate PBR materials, where both the specially designed multi-branch DiT and reference-based DiT blocks adopt a global attention mechanism to promote feature interaction and fusion between different views, thereby improving multi-view consistency. In addition, we adopt a PBR-based diffusion loss to ensure that the generated materials align with realistic physical principles. In the refinement stage, we propose a material-refined DiT that performs inpainting in empty areas and enhances details in UV space. Except for the normal condition, this refinement also takes the material map from the generation stage as an additional condition to reduce the learning difficulty and improve generalization. Extensive experiments show that our method achieves state-of-the-art performance in texturing 3D objects with PBR materials and provides significant advantages for graphics relighting applications. Project Page: https://lingtengqiu.github.io/2024/MCMat/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_14148 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MCMat: Multiview-Consistent and Physically Accurate PBR Material Generation Zhu, Shenhao Qiu, Lingteng Gu, Xiaodong Zhao, Zhengyi Xu, Chao He, Yuxiao Li, Zhe Han, Xiaoguang Yao, Yao Cao, Xun Zhu, Siyu Yuan, Weihao Dong, Zilong Zhu, Hao Computer Vision and Pattern Recognition Existing 2D methods utilize UNet-based diffusion models to generate multi-view physically-based rendering (PBR) maps but struggle with multi-view inconsistency, while some 3D methods directly generate UV maps, encountering generalization issues due to the limited 3D data. To address these problems, we propose a two-stage approach, including multi-view generation and UV materials refinement. In the generation stage, we adopt a Diffusion Transformer (DiT) model to generate PBR materials, where both the specially designed multi-branch DiT and reference-based DiT blocks adopt a global attention mechanism to promote feature interaction and fusion between different views, thereby improving multi-view consistency. In addition, we adopt a PBR-based diffusion loss to ensure that the generated materials align with realistic physical principles. In the refinement stage, we propose a material-refined DiT that performs inpainting in empty areas and enhances details in UV space. Except for the normal condition, this refinement also takes the material map from the generation stage as an additional condition to reduce the learning difficulty and improve generalization. Extensive experiments show that our method achieves state-of-the-art performance in texturing 3D objects with PBR materials and provides significant advantages for graphics relighting applications. Project Page: https://lingtengqiu.github.io/2024/MCMat/ |
| title | MCMat: Multiview-Consistent and Physically Accurate PBR Material Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.14148 |