Few-shot point cloud reconstruction and denoising via learned Guassian splats renderings and fine-tuned diffusion features
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917648395141120 |
|---|---|
| author | Bonazzi, Pietro Rakatosaona, Marie-Julie Cannici, Marco Tombari, Federico Scaramuzza, Davide |
| author_facet | Bonazzi, Pietro Rakatosaona, Marie-Julie Cannici, Marco Tombari, Federico Scaramuzza, Davide |
| contents | Existing deep learning methods for the reconstruction and denoising of point clouds rely on small datasets of 3D shapes. We circumvent the problem by leveraging deep learning methods trained on billions of images. We propose a method to reconstruct point clouds from few images and to denoise point clouds from their rendering by exploiting prior knowledge distilled from image-based deep learning models. To improve reconstruction in constraint settings, we regularize the training of a differentiable renderer with hybrid surface and appearance by introducing semantic consistency supervision. In addition, we propose a pipeline to finetune Stable Diffusion to denoise renderings of noisy point clouds and we demonstrate how these learned filters can be used to remove point cloud noise coming without 3D supervision. We compare our method with DSS and PointRadiance and achieved higher quality 3D reconstruction on the Sketchfab Testset and SCUT Dataset. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_01112 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Few-shot point cloud reconstruction and denoising via learned Guassian splats renderings and fine-tuned diffusion features Bonazzi, Pietro Rakatosaona, Marie-Julie Cannici, Marco Tombari, Federico Scaramuzza, Davide Computer Vision and Pattern Recognition Computational Geometry Existing deep learning methods for the reconstruction and denoising of point clouds rely on small datasets of 3D shapes. We circumvent the problem by leveraging deep learning methods trained on billions of images. We propose a method to reconstruct point clouds from few images and to denoise point clouds from their rendering by exploiting prior knowledge distilled from image-based deep learning models. To improve reconstruction in constraint settings, we regularize the training of a differentiable renderer with hybrid surface and appearance by introducing semantic consistency supervision. In addition, we propose a pipeline to finetune Stable Diffusion to denoise renderings of noisy point clouds and we demonstrate how these learned filters can be used to remove point cloud noise coming without 3D supervision. We compare our method with DSS and PointRadiance and achieved higher quality 3D reconstruction on the Sketchfab Testset and SCUT Dataset. |
| title | Few-shot point cloud reconstruction and denoising via learned Guassian splats renderings and fine-tuned diffusion features |
| topic | Computer Vision and Pattern Recognition Computational Geometry |
| url | https://arxiv.org/abs/2404.01112 |