DreamBeast: Distilling 3D Fantastical Animals with Part-Aware Knowledge Transfer
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910601840689152 |
|---|---|
| author | Li, Runjia Han, Junlin Melas-Kyriazi, Luke Sun, Chunyi An, Zhaochong Gui, Zhongrui Sun, Shuyang Torr, Philip Jakab, Tomas |
| author_facet | Li, Runjia Han, Junlin Melas-Kyriazi, Luke Sun, Chunyi An, Zhaochong Gui, Zhongrui Sun, Shuyang Torr, Philip Jakab, Tomas |
| contents | We present DreamBeast, a novel method based on score distillation sampling (SDS) for generating fantastical 3D animal assets composed of distinct parts. Existing SDS methods often struggle with this generation task due to a limited understanding of part-level semantics in text-to-image diffusion models. While recent diffusion models, such as Stable Diffusion 3, demonstrate a better part-level understanding, they are prohibitively slow and exhibit other common problems associated with single-view diffusion models. DreamBeast overcomes this limitation through a novel part-aware knowledge transfer mechanism. For each generated asset, we efficiently extract part-level knowledge from the Stable Diffusion 3 model into a 3D Part-Affinity implicit representation. This enables us to instantly generate Part-Affinity maps from arbitrary camera views, which we then use to modulate the guidance of a multi-view diffusion model during SDS to create 3D assets of fantastical animals. DreamBeast significantly enhances the quality of generated 3D creatures with user-specified part compositions while reducing computational overhead, as demonstrated by extensive quantitative and qualitative evaluations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_08271 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DreamBeast: Distilling 3D Fantastical Animals with Part-Aware Knowledge Transfer Li, Runjia Han, Junlin Melas-Kyriazi, Luke Sun, Chunyi An, Zhaochong Gui, Zhongrui Sun, Shuyang Torr, Philip Jakab, Tomas Computer Vision and Pattern Recognition Graphics Machine Learning Image and Video Processing We present DreamBeast, a novel method based on score distillation sampling (SDS) for generating fantastical 3D animal assets composed of distinct parts. Existing SDS methods often struggle with this generation task due to a limited understanding of part-level semantics in text-to-image diffusion models. While recent diffusion models, such as Stable Diffusion 3, demonstrate a better part-level understanding, they are prohibitively slow and exhibit other common problems associated with single-view diffusion models. DreamBeast overcomes this limitation through a novel part-aware knowledge transfer mechanism. For each generated asset, we efficiently extract part-level knowledge from the Stable Diffusion 3 model into a 3D Part-Affinity implicit representation. This enables us to instantly generate Part-Affinity maps from arbitrary camera views, which we then use to modulate the guidance of a multi-view diffusion model during SDS to create 3D assets of fantastical animals. DreamBeast significantly enhances the quality of generated 3D creatures with user-specified part compositions while reducing computational overhead, as demonstrated by extensive quantitative and qualitative evaluations. |
| title | DreamBeast: Distilling 3D Fantastical Animals with Part-Aware Knowledge Transfer |
| topic | Computer Vision and Pattern Recognition Graphics Machine Learning Image and Video Processing |
| url | https://arxiv.org/abs/2409.08271 |