USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913381892489216 |
|---|---|
| author | Wang, Xiaoqi He, Wenbin Xuan, Xiwei Sebastian, Clint Ono, Jorge Piazentin Li, Xin Behpour, Sima Doan, Thang Gou, Liang Shen, Han Wei Ren, Liu |
| author_facet | Wang, Xiaoqi He, Wenbin Xuan, Xiwei Sebastian, Clint Ono, Jorge Piazentin Li, Xin Behpour, Sima Doan, Thang Gou, Liang Shen, Han Wei Ren, Liu |
| contents | The open-vocabulary image segmentation task involves partitioning images into semantically meaningful segments and classifying them with flexible text-defined categories. The recent vision-based foundation models such as the Segment Anything Model (SAM) have shown superior performance in generating class-agnostic image segments. The main challenge in open-vocabulary image segmentation now lies in accurately classifying these segments into text-defined categories. In this paper, we introduce the Universal Segment Embedding (USE) framework to address this challenge. This framework is comprised of two key components: 1) a data pipeline designed to efficiently curate a large amount of segment-text pairs at various granularities, and 2) a universal segment embedding model that enables precise segment classification into a vast range of text-defined categories. The USE model can not only help open-vocabulary image segmentation but also facilitate other downstream tasks (e.g., querying and ranking). Through comprehensive experimental studies on semantic segmentation and part segmentation benchmarks, we demonstrate that the USE framework outperforms state-of-the-art open-vocabulary segmentation methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_05271 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentation Wang, Xiaoqi He, Wenbin Xuan, Xiwei Sebastian, Clint Ono, Jorge Piazentin Li, Xin Behpour, Sima Doan, Thang Gou, Liang Shen, Han Wei Ren, Liu Computer Vision and Pattern Recognition The open-vocabulary image segmentation task involves partitioning images into semantically meaningful segments and classifying them with flexible text-defined categories. The recent vision-based foundation models such as the Segment Anything Model (SAM) have shown superior performance in generating class-agnostic image segments. The main challenge in open-vocabulary image segmentation now lies in accurately classifying these segments into text-defined categories. In this paper, we introduce the Universal Segment Embedding (USE) framework to address this challenge. This framework is comprised of two key components: 1) a data pipeline designed to efficiently curate a large amount of segment-text pairs at various granularities, and 2) a universal segment embedding model that enables precise segment classification into a vast range of text-defined categories. The USE model can not only help open-vocabulary image segmentation but also facilitate other downstream tasks (e.g., querying and ranking). Through comprehensive experimental studies on semantic segmentation and part segmentation benchmarks, we demonstrate that the USE framework outperforms state-of-the-art open-vocabulary segmentation methods. |
| title | USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2406.05271 |