Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917985552171008 |
|---|---|
| author | Lee, Junha Park, Chunghyun Choe, Jaesung Wang, Yu-Chiang Frank Kautz, Jan Cho, Minsu Choy, Chris |
| author_facet | Lee, Junha Park, Chunghyun Choe, Jaesung Wang, Yu-Chiang Frank Kautz, Jan Cho, Minsu Choy, Chris |
| contents | We tackle open-vocabulary 3D scene understanding by introducing a novel data generation pipeline and training framework. Our method addresses three critical requirements for effective training: precise 3D region segmentation, comprehensive textual descriptions, and sufficient dataset scale. By leveraging state-of-the-art open-vocabulary image segmentation models and region-aware Vision-Language Models, we develop an automatic pipeline that generates high-quality 3D mask-text pairs. Applying this pipeline to multiple 3D scene datasets, we create Mosaic3D-5.6M, a dataset of over 30K annotated scenes with 5.6M mask-text pairs, significantly larger than existing datasets. Building upon this data, we propose Mosaic3D, a foundation model combining a 3D encoder trained with contrastive learning and a lightweight mask decoder for open-vocabulary 3D semantic and instance segmentation. Our approach achieves state-of-the-art results on open-vocabulary 3D semantic and instance segmentation tasks including ScanNet200, Matterport3D, and ScanNet++, with ablation studies validating the effectiveness of our large-scale training data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_02548 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentation Lee, Junha Park, Chunghyun Choe, Jaesung Wang, Yu-Chiang Frank Kautz, Jan Cho, Minsu Choy, Chris Computer Vision and Pattern Recognition We tackle open-vocabulary 3D scene understanding by introducing a novel data generation pipeline and training framework. Our method addresses three critical requirements for effective training: precise 3D region segmentation, comprehensive textual descriptions, and sufficient dataset scale. By leveraging state-of-the-art open-vocabulary image segmentation models and region-aware Vision-Language Models, we develop an automatic pipeline that generates high-quality 3D mask-text pairs. Applying this pipeline to multiple 3D scene datasets, we create Mosaic3D-5.6M, a dataset of over 30K annotated scenes with 5.6M mask-text pairs, significantly larger than existing datasets. Building upon this data, we propose Mosaic3D, a foundation model combining a 3D encoder trained with contrastive learning and a lightweight mask decoder for open-vocabulary 3D semantic and instance segmentation. Our approach achieves state-of-the-art results on open-vocabulary 3D semantic and instance segmentation tasks including ScanNet200, Matterport3D, and ScanNet++, with ablation studies validating the effectiveness of our large-scale training data. |
| title | Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2502.02548 |