InstantSplat: Sparse-view Gaussian Splatting in Seconds
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915370845077504 |
|---|---|
| author | Fan, Zhiwen Cong, Wenyan Wen, Kairun Wang, Kevin Zhang, Jian Ding, Xinghao Xu, Danfei Ivanovic, Boris Pavone, Marco Pavlakos, Georgios Wang, Zhangyang Wang, Yue |
| author_facet | Fan, Zhiwen Cong, Wenyan Wen, Kairun Wang, Kevin Zhang, Jian Ding, Xinghao Xu, Danfei Ivanovic, Boris Pavone, Marco Pavlakos, Georgios Wang, Zhangyang Wang, Yue |
| contents | While neural 3D reconstruction has advanced substantially, its performance significantly degrades with sparse-view data, which limits its broader applicability, since SfM is often unreliable in sparse-view scenarios where feature matches are scarce. In this paper, we introduce InstantSplat, a novel approach for addressing sparse-view 3D scene reconstruction at lightning-fast speed. InstantSplat employs a self-supervised framework that optimizes 3D scene representation and camera poses by unprojecting 2D pixels into 3D space and aligning them using differentiable neural rendering. The optimization process is initialized with a large-scale trained geometric foundation model, which provides dense priors that yield initial points through model inference, after which we further optimize all scene parameters using photometric errors. To mitigate redundancy introduced by the prior model, we propose a co-visibility-based geometry initialization, and a Gaussian-based bundle adjustment is employed to rapidly adapt both the scene representation and camera parameters without relying on a complex adaptive density control process. Overall, InstantSplat is compatible with multiple point-based representations for view synthesis and surface reconstruction. It achieves an acceleration of over 30x in reconstruction and improves visual quality (SSIM) from 0.3755 to 0.7624 compared to traditional SfM with 3D-GS. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_20309 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | InstantSplat: Sparse-view Gaussian Splatting in Seconds Fan, Zhiwen Cong, Wenyan Wen, Kairun Wang, Kevin Zhang, Jian Ding, Xinghao Xu, Danfei Ivanovic, Boris Pavone, Marco Pavlakos, Georgios Wang, Zhangyang Wang, Yue Computer Vision and Pattern Recognition While neural 3D reconstruction has advanced substantially, its performance significantly degrades with sparse-view data, which limits its broader applicability, since SfM is often unreliable in sparse-view scenarios where feature matches are scarce. In this paper, we introduce InstantSplat, a novel approach for addressing sparse-view 3D scene reconstruction at lightning-fast speed. InstantSplat employs a self-supervised framework that optimizes 3D scene representation and camera poses by unprojecting 2D pixels into 3D space and aligning them using differentiable neural rendering. The optimization process is initialized with a large-scale trained geometric foundation model, which provides dense priors that yield initial points through model inference, after which we further optimize all scene parameters using photometric errors. To mitigate redundancy introduced by the prior model, we propose a co-visibility-based geometry initialization, and a Gaussian-based bundle adjustment is employed to rapidly adapt both the scene representation and camera parameters without relying on a complex adaptive density control process. Overall, InstantSplat is compatible with multiple point-based representations for view synthesis and surface reconstruction. It achieves an acceleration of over 30x in reconstruction and improves visual quality (SSIM) from 0.3755 to 0.7624 compared to traditional SfM with 3D-GS. |
| title | InstantSplat: Sparse-view Gaussian Splatting in Seconds |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.20309 |