Details Matter for Indoor Open-vocabulary 3D Instance Segmentation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909712747855872 |
|---|---|
| author | Jung, Sanghun Zheng, Jingjing Zhang, Ke Qiao, Nan Chen, Albert Y. C. Xia, Lu Liu, Chi Sun, Yuyin Zeng, Xiao Huang, Hsiang-Wei Boots, Byron Sun, Min Kuo, Cheng-Hao |
| author_facet | Jung, Sanghun Zheng, Jingjing Zhang, Ke Qiao, Nan Chen, Albert Y. C. Xia, Lu Liu, Chi Sun, Yuyin Zeng, Xiao Huang, Hsiang-Wei Boots, Byron Sun, Min Kuo, Cheng-Hao |
| contents | Unlike closed-vocabulary 3D instance segmentation that is often trained end-to-end, open-vocabulary 3D instance segmentation (OV-3DIS) often leverages vision-language models (VLMs) to generate 3D instance proposals and classify them. While various concepts have been proposed from existing research, we observe that these individual concepts are not mutually exclusive but complementary. In this paper, we propose a new state-of-the-art solution for OV-3DIS by carefully designing a recipe to combine the concepts together and refining them to address key challenges. Our solution follows the two-stage scheme: 3D proposal generation and instance classification. We employ robust 3D tracking-based proposal aggregation to generate 3D proposals and remove overlapped or partial proposals by iterative merging/removal. For the classification stage, we replace the standard CLIP model with Alpha-CLIP, which incorporates object masks as an alpha channel to reduce background noise and obtain object-centric representation. Additionally, we introduce the standardized maximum similarity (SMS) score to normalize text-to-proposal similarity, effectively filtering out false positives and boosting precision. Our framework achieves state-of-the-art performance on ScanNet200 and S3DIS across all AP and AR metrics, even surpassing an end-to-end closed-vocabulary method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_23134 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Details Matter for Indoor Open-vocabulary 3D Instance Segmentation Jung, Sanghun Zheng, Jingjing Zhang, Ke Qiao, Nan Chen, Albert Y. C. Xia, Lu Liu, Chi Sun, Yuyin Zeng, Xiao Huang, Hsiang-Wei Boots, Byron Sun, Min Kuo, Cheng-Hao Computer Vision and Pattern Recognition Unlike closed-vocabulary 3D instance segmentation that is often trained end-to-end, open-vocabulary 3D instance segmentation (OV-3DIS) often leverages vision-language models (VLMs) to generate 3D instance proposals and classify them. While various concepts have been proposed from existing research, we observe that these individual concepts are not mutually exclusive but complementary. In this paper, we propose a new state-of-the-art solution for OV-3DIS by carefully designing a recipe to combine the concepts together and refining them to address key challenges. Our solution follows the two-stage scheme: 3D proposal generation and instance classification. We employ robust 3D tracking-based proposal aggregation to generate 3D proposals and remove overlapped or partial proposals by iterative merging/removal. For the classification stage, we replace the standard CLIP model with Alpha-CLIP, which incorporates object masks as an alpha channel to reduce background noise and obtain object-centric representation. Additionally, we introduce the standardized maximum similarity (SMS) score to normalize text-to-proposal similarity, effectively filtering out false positives and boosting precision. Our framework achieves state-of-the-art performance on ScanNet200 and S3DIS across all AP and AR metrics, even surpassing an end-to-end closed-vocabulary method. |
| title | Details Matter for Indoor Open-vocabulary 3D Instance Segmentation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2507.23134 |