kNN-CLIP: Retrieval Enables Training-Free Segmentation on Continually Expanding Large Vocabularies
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909285635588096 |
|---|---|
| author | Gui, Zhongrui Sun, Shuyang Li, Runjia Yuan, Jianhao An, Zhaochong Roth, Karsten Prabhu, Ameya Torr, Philip |
| author_facet | Gui, Zhongrui Sun, Shuyang Li, Runjia Yuan, Jianhao An, Zhaochong Roth, Karsten Prabhu, Ameya Torr, Philip |
| contents | Continual segmentation has not yet tackled the challenge of improving open-vocabulary segmentation models with training data for accurate segmentation across large, continually expanding vocabularies. We discover that traditional continual training results in severe catastrophic forgetting, failing to outperform a zero-shot segmentation baseline. We introduce a novel training-free strategy, kNN-CLIP, which augments the model with a database of instance embeddings for semantic and panoptic segmentation that achieves zero forgetting. We demonstrate that kNN-CLIP can adapt to continually growing vocabularies without the need for retraining or large memory costs. kNN-CLIP enables open-vocabulary segmentation methods to expand their vocabularies on any domain with a single pass through the data, while only storing compact embeddings. This approach minimizes both compute and memory costs. kNN-CLIP achieves state-of-the-art performance across large-vocabulary semantic and panoptic segmentation datasets. We hope kNN-CLIP represents a significant step forward in enabling more efficient and adaptable continual segmentation, paving the way for advances in real-world large-vocabulary continual segmentation methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_09447 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | kNN-CLIP: Retrieval Enables Training-Free Segmentation on Continually Expanding Large Vocabularies Gui, Zhongrui Sun, Shuyang Li, Runjia Yuan, Jianhao An, Zhaochong Roth, Karsten Prabhu, Ameya Torr, Philip Computer Vision and Pattern Recognition Machine Learning Continual segmentation has not yet tackled the challenge of improving open-vocabulary segmentation models with training data for accurate segmentation across large, continually expanding vocabularies. We discover that traditional continual training results in severe catastrophic forgetting, failing to outperform a zero-shot segmentation baseline. We introduce a novel training-free strategy, kNN-CLIP, which augments the model with a database of instance embeddings for semantic and panoptic segmentation that achieves zero forgetting. We demonstrate that kNN-CLIP can adapt to continually growing vocabularies without the need for retraining or large memory costs. kNN-CLIP enables open-vocabulary segmentation methods to expand their vocabularies on any domain with a single pass through the data, while only storing compact embeddings. This approach minimizes both compute and memory costs. kNN-CLIP achieves state-of-the-art performance across large-vocabulary semantic and panoptic segmentation datasets. We hope kNN-CLIP represents a significant step forward in enabling more efficient and adaptable continual segmentation, paving the way for advances in real-world large-vocabulary continual segmentation methods. |
| title | kNN-CLIP: Retrieval Enables Training-Free Segmentation on Continually Expanding Large Vocabularies |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2404.09447 |