ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914890214539264 |
|---|---|
| author | Höllein, Lukas Božič, Aljaž Müller, Norman Novotny, David Tseng, Hung-Yu Richardt, Christian Zollhöfer, Michael Nießner, Matthias |
| author_facet | Höllein, Lukas Božič, Aljaž Müller, Norman Novotny, David Tseng, Hung-Yu Richardt, Christian Zollhöfer, Michael Nießner, Matthias |
| contents | 3D asset generation is getting massive amounts of attention, inspired by the recent success of text-guided 2D content creation. Existing text-to-3D methods use pretrained text-to-image diffusion models in an optimization problem or fine-tune them on synthetic data, which often results in non-photorealistic 3D objects without backgrounds. In this paper, we present a method that leverages pretrained text-to-image models as a prior, and learn to generate multi-view images in a single denoising process from real-world data. Concretely, we propose to integrate 3D volume-rendering and cross-frame-attention layers into each block of the existing U-Net network of the text-to-image model. Moreover, we design an autoregressive generation that renders more 3D-consistent images at any viewpoint. We train our model on real-world datasets of objects and showcase its capabilities to generate instances with a variety of high-quality shapes and textures in authentic surroundings. Compared to the existing methods, the results generated by our method are consistent, and have favorable visual quality (-30% FID, -37% KID). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_01807 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models Höllein, Lukas Božič, Aljaž Müller, Norman Novotny, David Tseng, Hung-Yu Richardt, Christian Zollhöfer, Michael Nießner, Matthias Computer Vision and Pattern Recognition 3D asset generation is getting massive amounts of attention, inspired by the recent success of text-guided 2D content creation. Existing text-to-3D methods use pretrained text-to-image diffusion models in an optimization problem or fine-tune them on synthetic data, which often results in non-photorealistic 3D objects without backgrounds. In this paper, we present a method that leverages pretrained text-to-image models as a prior, and learn to generate multi-view images in a single denoising process from real-world data. Concretely, we propose to integrate 3D volume-rendering and cross-frame-attention layers into each block of the existing U-Net network of the text-to-image model. Moreover, we design an autoregressive generation that renders more 3D-consistent images at any viewpoint. We train our model on real-world datasets of objects and showcase its capabilities to generate instances with a variety of high-quality shapes and textures in authentic surroundings. Compared to the existing methods, the results generated by our method are consistent, and have favorable visual quality (-30% FID, -37% KID). |
| title | ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.01807 |