TextRegion: Text-Aligned Region Tokens from Frozen Image-Text Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915602457690112 |
|---|---|
| author | Xiao, Yao Fu, Qiqian Tao, Heyi Wu, Yuqun Zhu, Zhen Hoiem, Derek |
| author_facet | Xiao, Yao Fu, Qiqian Tao, Heyi Wu, Yuqun Zhu, Zhen Hoiem, Derek |
| contents | Image-text models excel at image-level tasks but struggle with detailed visual understanding. While these models provide strong visual-language alignment, segmentation models like SAM2 offer precise spatial boundaries for objects. To this end, we propose TextRegion, a simple, effective, and training-free framework that combines the strengths of image-text models and SAM2 to generate powerful text-aligned region tokens. These tokens enable detailed visual understanding while preserving open-vocabulary capabilities. They can be directly applied to various downstream tasks, including open-world semantic segmentation, referring expression comprehension, and grounding. We conduct extensive evaluations and consistently achieve superior or competitive performance compared to state-of-the-art training-free methods. Additionally, our framework is compatible with many image-text models, making it highly practical and easily extensible as stronger models emerge. Code is available at: https://github.com/avaxiao/TextRegion. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_23769 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | TextRegion: Text-Aligned Region Tokens from Frozen Image-Text Models Xiao, Yao Fu, Qiqian Tao, Heyi Wu, Yuqun Zhu, Zhen Hoiem, Derek Computer Vision and Pattern Recognition Image-text models excel at image-level tasks but struggle with detailed visual understanding. While these models provide strong visual-language alignment, segmentation models like SAM2 offer precise spatial boundaries for objects. To this end, we propose TextRegion, a simple, effective, and training-free framework that combines the strengths of image-text models and SAM2 to generate powerful text-aligned region tokens. These tokens enable detailed visual understanding while preserving open-vocabulary capabilities. They can be directly applied to various downstream tasks, including open-world semantic segmentation, referring expression comprehension, and grounding. We conduct extensive evaluations and consistently achieve superior or competitive performance compared to state-of-the-art training-free methods. Additionally, our framework is compatible with many image-text models, making it highly practical and easily extensible as stronger models emerge. Code is available at: https://github.com/avaxiao/TextRegion. |
| title | TextRegion: Text-Aligned Region Tokens from Frozen Image-Text Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2505.23769 |