Effective SAM Combination for Open-Vocabulary Semantic Segmentation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909556621180928 |
|---|---|
| author | Lee, Minhyeok Cho, Suhwan Lee, Jungho Yang, Sunghun Choi, Heeseung Kim, Ig-Jae Lee, Sangyoun |
| author_facet | Lee, Minhyeok Cho, Suhwan Lee, Jungho Yang, Sunghun Choi, Heeseung Kim, Ig-Jae Lee, Sangyoun |
| contents | Open-vocabulary semantic segmentation aims to assign pixel-level labels to images across an unlimited range of classes. Traditional methods address this by sequentially connecting a powerful mask proposal generator, such as the Segment Anything Model (SAM), with a pre-trained vision-language model like CLIP. But these two-stage approaches often suffer from high computational costs, memory inefficiencies. In this paper, we propose ESC-Net, a novel one-stage open-vocabulary segmentation model that leverages the SAM decoder blocks for class-agnostic segmentation within an efficient inference framework. By embedding pseudo prompts generated from image-text correlations into SAM's promptable segmentation framework, ESC-Net achieves refined spatial aggregation for accurate mask predictions. ESC-Net achieves superior performance on standard benchmarks, including ADE20K, PASCAL-VOC, and PASCAL-Context, outperforming prior methods in both efficiency and accuracy. Comprehensive ablation studies further demonstrate its robustness across challenging conditions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_14723 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Effective SAM Combination for Open-Vocabulary Semantic Segmentation Lee, Minhyeok Cho, Suhwan Lee, Jungho Yang, Sunghun Choi, Heeseung Kim, Ig-Jae Lee, Sangyoun Computer Vision and Pattern Recognition Open-vocabulary semantic segmentation aims to assign pixel-level labels to images across an unlimited range of classes. Traditional methods address this by sequentially connecting a powerful mask proposal generator, such as the Segment Anything Model (SAM), with a pre-trained vision-language model like CLIP. But these two-stage approaches often suffer from high computational costs, memory inefficiencies. In this paper, we propose ESC-Net, a novel one-stage open-vocabulary segmentation model that leverages the SAM decoder blocks for class-agnostic segmentation within an efficient inference framework. By embedding pseudo prompts generated from image-text correlations into SAM's promptable segmentation framework, ESC-Net achieves refined spatial aggregation for accurate mask predictions. ESC-Net achieves superior performance on standard benchmarks, including ADE20K, PASCAL-VOC, and PASCAL-Context, outperforming prior methods in both efficiency and accuracy. Comprehensive ablation studies further demonstrate its robustness across challenging conditions. |
| title | Effective SAM Combination for Open-Vocabulary Semantic Segmentation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2411.14723 |