SPAD : Spatially Aware Multiview Diffusers
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929237718466560 |
|---|---|
| author | Kant, Yash Wu, Ziyi Vasilkovsky, Michael Qian, Guocheng Ren, Jian Guler, Riza Alp Ghanem, Bernard Tulyakov, Sergey Gilitschenski, Igor Siarohin, Aliaksandr |
| author_facet | Kant, Yash Wu, Ziyi Vasilkovsky, Michael Qian, Guocheng Ren, Jian Guler, Riza Alp Ghanem, Bernard Tulyakov, Sergey Gilitschenski, Igor Siarohin, Aliaksandr |
| contents | We present SPAD, a novel approach for creating consistent multi-view images from text prompts or single images. To enable multi-view generation, we repurpose a pretrained 2D diffusion model by extending its self-attention layers with cross-view interactions, and fine-tune it on a high quality subset of Objaverse. We find that a naive extension of the self-attention proposed in prior work (e.g. MVDream) leads to content copying between views. Therefore, we explicitly constrain the cross-view attention based on epipolar geometry. To further enhance 3D consistency, we utilize Plucker coordinates derived from camera rays and inject them as positional encoding. This enables SPAD to reason over spatial proximity in 3D well. In contrast to recent works that can only generate views at fixed azimuth and elevation, SPAD offers full camera control and achieves state-of-the-art results in novel view synthesis on unseen objects from the Objaverse and Google Scanned Objects datasets. Finally, we demonstrate that text-to-3D generation using SPAD prevents the multi-face Janus issue. See more details at our webpage: https://yashkant.github.io/spad |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_05235 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SPAD : Spatially Aware Multiview Diffusers Kant, Yash Wu, Ziyi Vasilkovsky, Michael Qian, Guocheng Ren, Jian Guler, Riza Alp Ghanem, Bernard Tulyakov, Sergey Gilitschenski, Igor Siarohin, Aliaksandr Computer Vision and Pattern Recognition We present SPAD, a novel approach for creating consistent multi-view images from text prompts or single images. To enable multi-view generation, we repurpose a pretrained 2D diffusion model by extending its self-attention layers with cross-view interactions, and fine-tune it on a high quality subset of Objaverse. We find that a naive extension of the self-attention proposed in prior work (e.g. MVDream) leads to content copying between views. Therefore, we explicitly constrain the cross-view attention based on epipolar geometry. To further enhance 3D consistency, we utilize Plucker coordinates derived from camera rays and inject them as positional encoding. This enables SPAD to reason over spatial proximity in 3D well. In contrast to recent works that can only generate views at fixed azimuth and elevation, SPAD offers full camera control and achieves state-of-the-art results in novel view synthesis on unseen objects from the Objaverse and Google Scanned Objects datasets. Finally, we demonstrate that text-to-3D generation using SPAD prevents the multi-face Janus issue. See more details at our webpage: https://yashkant.github.io/spad |
| title | SPAD : Spatially Aware Multiview Diffusers |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2402.05235 |