MaPa: Text-driven Photorealistic Material Painting for 3D Shapes
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913790150311936 |
|---|---|
| author | Zhang, Shangzan Peng, Sida Xu, Tao Yang, Yuanbo Chen, Tianrun Xue, Nan Shen, Yujun Bao, Hujun Hu, Ruizhen Zhou, Xiaowei |
| author_facet | Zhang, Shangzan Peng, Sida Xu, Tao Yang, Yuanbo Chen, Tianrun Xue, Nan Shen, Yujun Bao, Hujun Hu, Ruizhen Zhou, Xiaowei |
| contents | This paper aims to generate materials for 3D meshes from text descriptions. Unlike existing methods that synthesize texture maps, we propose to generate segment-wise procedural material graphs as the appearance representation, which supports high-quality rendering and provides substantial flexibility in editing. Instead of relying on extensive paired data, i.e., 3D meshes with material graphs and corresponding text descriptions, to train a material graph generative model, we propose to leverage the pre-trained 2D diffusion model as a bridge to connect the text and material graphs. Specifically, our approach decomposes a shape into a set of segments and designs a segment-controlled diffusion model to synthesize 2D images that are aligned with mesh parts. Based on generated images, we initialize parameters of material graphs and fine-tune them through the differentiable rendering module to produce materials in accordance with the textual description. Extensive experiments demonstrate the superior performance of our framework in photorealism, resolution, and editability over existing methods. Project page: https://zju3dv.github.io/MaPa |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_17569 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MaPa: Text-driven Photorealistic Material Painting for 3D Shapes Zhang, Shangzan Peng, Sida Xu, Tao Yang, Yuanbo Chen, Tianrun Xue, Nan Shen, Yujun Bao, Hujun Hu, Ruizhen Zhou, Xiaowei Computer Vision and Pattern Recognition This paper aims to generate materials for 3D meshes from text descriptions. Unlike existing methods that synthesize texture maps, we propose to generate segment-wise procedural material graphs as the appearance representation, which supports high-quality rendering and provides substantial flexibility in editing. Instead of relying on extensive paired data, i.e., 3D meshes with material graphs and corresponding text descriptions, to train a material graph generative model, we propose to leverage the pre-trained 2D diffusion model as a bridge to connect the text and material graphs. Specifically, our approach decomposes a shape into a set of segments and designs a segment-controlled diffusion model to synthesize 2D images that are aligned with mesh parts. Based on generated images, we initialize parameters of material graphs and fine-tune them through the differentiable rendering module to produce materials in accordance with the textual description. Extensive experiments demonstrate the superior performance of our framework in photorealism, resolution, and editability over existing methods. Project page: https://zju3dv.github.io/MaPa |
| title | MaPa: Text-driven Photorealistic Material Painting for 3D Shapes |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2404.17569 |