SC-Diff: 3D Shape Completion with Latent Diffusion Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914014617927680 |
|---|---|
| author | Schaefer, Simon Galvis, Juan D. Zuo, Xingxing Leutengger, Stefan |
| author_facet | Schaefer, Simon Galvis, Juan D. Zuo, Xingxing Leutengger, Stefan |
| contents | We present a novel 3D shape completion framework that unifies multimodal conditioning, leveraging both 2D images and 3D partial scans through a latent diffusion model. Shapes are represented as Truncated Signed Distance Functions (TSDFs) and encoded into a discrete latent space jointly supervised by 2D and 3D cues, enabling efficient high-resolution processing while reducing GPU memory usage by 30\% compared to state-of-the-art methods. Our approach guides the generation process with flexible multimodal conditioning, ensuring consistent integration of 2D and 3D information from encoding to reconstruction. Our training strategy simulates realistic partial observations, avoiding assumptions about input structure and improving robustness in real-world scenarios. Leveraging our efficient latent space and multimodal conditioning, our model generalizes across object categories, outperforming class-specific models by 12\% and class-agnostic models by 47\% in $l_1$ reconstruction error, while producing more diverse, realistic, and high-fidelity completions than prior approaches. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_12470 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SC-Diff: 3D Shape Completion with Latent Diffusion Models Schaefer, Simon Galvis, Juan D. Zuo, Xingxing Leutengger, Stefan Computer Vision and Pattern Recognition We present a novel 3D shape completion framework that unifies multimodal conditioning, leveraging both 2D images and 3D partial scans through a latent diffusion model. Shapes are represented as Truncated Signed Distance Functions (TSDFs) and encoded into a discrete latent space jointly supervised by 2D and 3D cues, enabling efficient high-resolution processing while reducing GPU memory usage by 30\% compared to state-of-the-art methods. Our approach guides the generation process with flexible multimodal conditioning, ensuring consistent integration of 2D and 3D information from encoding to reconstruction. Our training strategy simulates realistic partial observations, avoiding assumptions about input structure and improving robustness in real-world scenarios. Leveraging our efficient latent space and multimodal conditioning, our model generalizes across object categories, outperforming class-specific models by 12\% and class-agnostic models by 47\% in $l_1$ reconstruction error, while producing more diverse, realistic, and high-fidelity completions than prior approaches. |
| title | SC-Diff: 3D Shape Completion with Latent Diffusion Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.12470 |