UniDream: Unifying Diffusion Priors for Relightable Text-to-3D Generation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910525771743232 |
|---|---|
| author | Liu, Zexiang Li, Yangguang Lin, Youtian Yu, Xin Peng, Sida Cao, Yan-Pei Qi, Xiaojuan Huang, Xiaoshui Liang, Ding Ouyang, Wanli |
| author_facet | Liu, Zexiang Li, Yangguang Lin, Youtian Yu, Xin Peng, Sida Cao, Yan-Pei Qi, Xiaojuan Huang, Xiaoshui Liang, Ding Ouyang, Wanli |
| contents | Recent advancements in text-to-3D generation technology have significantly advanced the conversion of textual descriptions into imaginative well-geometrical and finely textured 3D objects. Despite these developments, a prevalent limitation arises from the use of RGB data in diffusion or reconstruction models, which often results in models with inherent lighting and shadows effects that detract from their realism, thereby limiting their usability in applications that demand accurate relighting capabilities. To bridge this gap, we present UniDream, a text-to-3D generation framework by incorporating unified diffusion priors. Our approach consists of three main components: (1) a dual-phase training process to get albedo-normal aligned multi-view diffusion and reconstruction models, (2) a progressive generation procedure for geometry and albedo-textures based on Score Distillation Sample (SDS) using the trained reconstruction and diffusion models, and (3) an innovative application of SDS for finalizing PBR generation while keeping a fixed albedo based on Stable Diffusion model. Extensive evaluations demonstrate that UniDream surpasses existing methods in generating 3D objects with clearer albedo textures, smoother surfaces, enhanced realism, and superior relighting capabilities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_08754 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | UniDream: Unifying Diffusion Priors for Relightable Text-to-3D Generation Liu, Zexiang Li, Yangguang Lin, Youtian Yu, Xin Peng, Sida Cao, Yan-Pei Qi, Xiaojuan Huang, Xiaoshui Liang, Ding Ouyang, Wanli Computer Vision and Pattern Recognition Recent advancements in text-to-3D generation technology have significantly advanced the conversion of textual descriptions into imaginative well-geometrical and finely textured 3D objects. Despite these developments, a prevalent limitation arises from the use of RGB data in diffusion or reconstruction models, which often results in models with inherent lighting and shadows effects that detract from their realism, thereby limiting their usability in applications that demand accurate relighting capabilities. To bridge this gap, we present UniDream, a text-to-3D generation framework by incorporating unified diffusion priors. Our approach consists of three main components: (1) a dual-phase training process to get albedo-normal aligned multi-view diffusion and reconstruction models, (2) a progressive generation procedure for geometry and albedo-textures based on Score Distillation Sample (SDS) using the trained reconstruction and diffusion models, and (3) an innovative application of SDS for finalizing PBR generation while keeping a fixed albedo based on Stable Diffusion model. Extensive evaluations demonstrate that UniDream surpasses existing methods in generating 3D objects with clearer albedo textures, smoother surfaces, enhanced realism, and superior relighting capabilities. |
| title | UniDream: Unifying Diffusion Priors for Relightable Text-to-3D Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2312.08754 |