3D Open-Vocabulary Panoptic Segmentation with 2D-3D Vision-Language Distillation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909158830243840 |
|---|---|
| author | Xiao, Zihao Jing, Longlong Wu, Shangxuan Zhu, Alex Zihao Ji, Jingwei Jiang, Chiyu Max Hung, Wei-Chih Funkhouser, Thomas Kuo, Weicheng Angelova, Anelia Zhou, Yin Sheng, Shiwei |
| author_facet | Xiao, Zihao Jing, Longlong Wu, Shangxuan Zhu, Alex Zihao Ji, Jingwei Jiang, Chiyu Max Hung, Wei-Chih Funkhouser, Thomas Kuo, Weicheng Angelova, Anelia Zhou, Yin Sheng, Shiwei |
| contents | 3D panoptic segmentation is a challenging perception task, especially in autonomous driving. It aims to predict both semantic and instance annotations for 3D points in a scene. Although prior 3D panoptic segmentation approaches have achieved great performance on closed-set benchmarks, generalizing these approaches to unseen things and unseen stuff categories remains an open problem. For unseen object categories, 2D open-vocabulary segmentation has achieved promising results that solely rely on frozen CLIP backbones and ensembling multiple classification outputs. However, we find that simply extending these 2D models to 3D does not guarantee good performance due to poor per-mask classification quality, especially for novel stuff categories. In this paper, we propose the first method to tackle 3D open-vocabulary panoptic segmentation. Our model takes advantage of the fusion between learnable LiDAR features and dense frozen vision CLIP features, using a single classification head to make predictions for both base and novel classes. To further improve the classification performance on novel classes and leverage the CLIP model, we propose two novel loss functions: object-level distillation loss and voxel-level distillation loss. Our experiments on the nuScenes and SemanticKITTI datasets show that our method outperforms the strong baseline by a large margin. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_02402 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | 3D Open-Vocabulary Panoptic Segmentation with 2D-3D Vision-Language Distillation Xiao, Zihao Jing, Longlong Wu, Shangxuan Zhu, Alex Zihao Ji, Jingwei Jiang, Chiyu Max Hung, Wei-Chih Funkhouser, Thomas Kuo, Weicheng Angelova, Anelia Zhou, Yin Sheng, Shiwei Computer Vision and Pattern Recognition 3D panoptic segmentation is a challenging perception task, especially in autonomous driving. It aims to predict both semantic and instance annotations for 3D points in a scene. Although prior 3D panoptic segmentation approaches have achieved great performance on closed-set benchmarks, generalizing these approaches to unseen things and unseen stuff categories remains an open problem. For unseen object categories, 2D open-vocabulary segmentation has achieved promising results that solely rely on frozen CLIP backbones and ensembling multiple classification outputs. However, we find that simply extending these 2D models to 3D does not guarantee good performance due to poor per-mask classification quality, especially for novel stuff categories. In this paper, we propose the first method to tackle 3D open-vocabulary panoptic segmentation. Our model takes advantage of the fusion between learnable LiDAR features and dense frozen vision CLIP features, using a single classification head to make predictions for both base and novel classes. To further improve the classification performance on novel classes and leverage the CLIP model, we propose two novel loss functions: object-level distillation loss and voxel-level distillation loss. Our experiments on the nuScenes and SemanticKITTI datasets show that our method outperforms the strong baseline by a large margin. |
| title | 3D Open-Vocabulary Panoptic Segmentation with 2D-3D Vision-Language Distillation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2401.02402 |