Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2407.13642 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910533902401536 |
|---|---|
| author | Zhu, Xiaoyu Zhou, Hao Xing, Pengfei Zhao, Long Xu, Hao Liang, Junwei Hauptmann, Alexander Liu, Ting Gallagher, Andrew |
| author_facet | Zhu, Xiaoyu Zhou, Hao Xing, Pengfei Zhao, Long Xu, Hao Liang, Junwei Hauptmann, Alexander Liu, Ting Gallagher, Andrew |
| contents | In this paper, we investigate the use of diffusion models which are pre-trained on large-scale image-caption pairs for open-vocabulary 3D semantic understanding. We propose a novel method, namely Diff2Scene, which leverages frozen representations from text-image generative models, along with salient-aware and geometric-aware masks, for open-vocabulary 3D semantic segmentation and visual grounding tasks. Diff2Scene gets rid of any labeled 3D data and effectively identifies objects, appearances, materials, locations and their compositions in 3D scenes. We show that it outperforms competitive baselines and achieves significant improvements over state-of-the-art methods. In particular, Diff2Scene improves the state-of-the-art method on ScanNet200 by 12%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_13642 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Models Zhu, Xiaoyu Zhou, Hao Xing, Pengfei Zhao, Long Xu, Hao Liang, Junwei Hauptmann, Alexander Liu, Ting Gallagher, Andrew Computer Vision and Pattern Recognition In this paper, we investigate the use of diffusion models which are pre-trained on large-scale image-caption pairs for open-vocabulary 3D semantic understanding. We propose a novel method, namely Diff2Scene, which leverages frozen representations from text-image generative models, along with salient-aware and geometric-aware masks, for open-vocabulary 3D semantic segmentation and visual grounding tasks. Diff2Scene gets rid of any labeled 3D data and effectively identifies objects, appearances, materials, locations and their compositions in 3D scenes. We show that it outperforms competitive baselines and achieves significant improvements over state-of-the-art methods. In particular, Diff2Scene improves the state-of-the-art method on ScanNet200 by 12%. |
| title | Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.13642 |