Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929303959109632 |
|---|---|
| author | Nguyen, Phuc D. A. Ngo, Tuan Duc Kalogerakis, Evangelos Gan, Chuang Tran, Anh Pham, Cuong Nguyen, Khoi |
| author_facet | Nguyen, Phuc D. A. Ngo, Tuan Duc Kalogerakis, Evangelos Gan, Chuang Tran, Anh Pham, Cuong Nguyen, Khoi |
| contents | We introduce Open3DIS, a novel solution designed to tackle the problem of Open-Vocabulary Instance Segmentation within 3D scenes. Objects within 3D environments exhibit diverse shapes, scales, and colors, making precise instance-level identification a challenging task. Recent advancements in Open-Vocabulary scene understanding have made significant strides in this area by employing class-agnostic 3D instance proposal networks for object localization and learning queryable features for each 3D mask. While these methods produce high-quality instance proposals, they struggle with identifying small-scale and geometrically ambiguous objects. The key idea of our method is a new module that aggregates 2D instance masks across frames and maps them to geometrically coherent point cloud regions as high-quality object proposals addressing the above limitations. These are then combined with 3D class-agnostic instance proposals to include a wide range of objects in the real world. To validate our approach, we conducted experiments on three prominent datasets, including ScanNet200, S3DIS, and Replica, demonstrating significant performance gains in segmenting objects with diverse categories over the state-of-the-art approaches. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_10671 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance Nguyen, Phuc D. A. Ngo, Tuan Duc Kalogerakis, Evangelos Gan, Chuang Tran, Anh Pham, Cuong Nguyen, Khoi Computer Vision and Pattern Recognition We introduce Open3DIS, a novel solution designed to tackle the problem of Open-Vocabulary Instance Segmentation within 3D scenes. Objects within 3D environments exhibit diverse shapes, scales, and colors, making precise instance-level identification a challenging task. Recent advancements in Open-Vocabulary scene understanding have made significant strides in this area by employing class-agnostic 3D instance proposal networks for object localization and learning queryable features for each 3D mask. While these methods produce high-quality instance proposals, they struggle with identifying small-scale and geometrically ambiguous objects. The key idea of our method is a new module that aggregates 2D instance masks across frames and maps them to geometrically coherent point cloud regions as high-quality object proposals addressing the above limitations. These are then combined with 3D class-agnostic instance proposals to include a wide range of objects in the real world. To validate our approach, we conducted experiments on three prominent datasets, including ScanNet200, S3DIS, and Replica, demonstrating significant performance gains in segmenting objects with diverse categories over the state-of-the-art approaches. |
| title | Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2312.10671 |