Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2412.10231 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914266806747136 |
|---|---|
| author | Liang, Siyun Wang, Sen Li, Kunyi Niemeyer, Michael Gasperini, Stefano Lensch, Hendrik P. A. Navab, Nassir Tombari, Federico |
| author_facet | Liang, Siyun Wang, Sen Li, Kunyi Niemeyer, Michael Gasperini, Stefano Lensch, Hendrik P. A. Navab, Nassir Tombari, Federico |
| contents | 3D Gaussian Splatting has recently gained traction for its efficient training and real-time rendering. While its vanilla representation is mainly designed for view synthesis, recent works extended it to scene understanding with language features. However, storing additional high-dimensional features per Gaussian for semantic information is memory-intensive, which limits their ability to segment and interpret challenging scenes. To this end, we introduce SuperGSeg, a novel approach that fosters cohesive, context-aware hierarchical scene representation by disentangling segmentation and language field distillation. SuperGSeg first employs neural 3D Gaussians to learn geometry, instance and hierarchical segmentation features from multi-view images with the aid of off-the-shelf 2D masks. These features are then leveraged to create a sparse set of \acrlong{superg}s. \acrlong{superg}s facilitate the lifting and distillation of 2D language features into 3D space. They enable hierarchical scene understanding with high-dimensional language feature rendering at moderate GPU memory costs. Extensive experiments demonstrate that SuperGSeg achieves remarkable performance on both open-vocabulary object selection and semantic segmentation tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_10231 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SuperGSeg: Open-Vocabulary 3D Segmentation with Structured Super-Gaussians Liang, Siyun Wang, Sen Li, Kunyi Niemeyer, Michael Gasperini, Stefano Lensch, Hendrik P. A. Navab, Nassir Tombari, Federico Computer Vision and Pattern Recognition 3D Gaussian Splatting has recently gained traction for its efficient training and real-time rendering. While its vanilla representation is mainly designed for view synthesis, recent works extended it to scene understanding with language features. However, storing additional high-dimensional features per Gaussian for semantic information is memory-intensive, which limits their ability to segment and interpret challenging scenes. To this end, we introduce SuperGSeg, a novel approach that fosters cohesive, context-aware hierarchical scene representation by disentangling segmentation and language field distillation. SuperGSeg first employs neural 3D Gaussians to learn geometry, instance and hierarchical segmentation features from multi-view images with the aid of off-the-shelf 2D masks. These features are then leveraged to create a sparse set of \acrlong{superg}s. \acrlong{superg}s facilitate the lifting and distillation of 2D language features into 3D space. They enable hierarchical scene understanding with high-dimensional language feature rendering at moderate GPU memory costs. Extensive experiments demonstrate that SuperGSeg achieves remarkable performance on both open-vocabulary object selection and semantic segmentation tasks. |
| title | SuperGSeg: Open-Vocabulary 3D Segmentation with Structured Super-Gaussians |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.10231 |