Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.09480 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916687882747904 |
|---|---|
| author | Feng, Yongchao Liu, Yajie Yang, Shuai Cai, Wenrui Zhang, Jinqing Zhan, Qiqi Huang, Ziyue Yan, Hongxi Wan, Qiao Liu, Chenguang Wang, Junzhe Lv, Jiahui Liu, Ziqi Shi, Tengyuan Liu, Qingjie Wang, Yunhong |
| author_facet | Feng, Yongchao Liu, Yajie Yang, Shuai Cai, Wenrui Zhang, Jinqing Zhan, Qiqi Huang, Ziyue Yan, Hongxi Wan, Qiao Liu, Chenguang Wang, Junzhe Lv, Jiahui Liu, Ziqi Shi, Tengyuan Liu, Qingjie Wang, Yunhong |
| contents | Vision-Language Model (VLM) have gained widespread adoption in Open-Vocabulary (OV) object detection and segmentation tasks. Despite they have shown promise on OV-related tasks, their effectiveness in conventional vision tasks has thus far been unevaluated. In this work, we present the systematic review of VLM-based detection and segmentation, view VLM as the foundational model and conduct comprehensive evaluations across multiple downstream tasks for the first time: 1) The evaluation spans eight detection scenarios (closed-set detection, domain adaptation, crowded objects, etc.) and eight segmentation scenarios (few-shot, open-world, small object, etc.), revealing distinct performance advantages and limitations of various VLM architectures across tasks. 2) As for detection tasks, we evaluate VLMs under three finetuning granularities: \textit{zero prediction}, \textit{visual fine-tuning}, and \textit{text prompt}, and further analyze how different finetuning strategies impact performance under varied task. 3) Based on empirical findings, we provide in-depth analysis of the correlations between task characteristics, model architectures, and training methodologies, offering insights for future VLM design. 4) We believe that this work shall be valuable to the pattern recognition experts working in the fields of computer vision, multimodal learning, and vision foundation models by introducing them to the problem, and familiarizing them with the current status of the progress while providing promising directions for future research. A project associated with this review and evaluation has been created at https://github.com/better-chao/perceptual_abilities_evaluation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_09480 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Vision-Language Model for Object Detection and Segmentation: A Review and Evaluation Feng, Yongchao Liu, Yajie Yang, Shuai Cai, Wenrui Zhang, Jinqing Zhan, Qiqi Huang, Ziyue Yan, Hongxi Wan, Qiao Liu, Chenguang Wang, Junzhe Lv, Jiahui Liu, Ziqi Shi, Tengyuan Liu, Qingjie Wang, Yunhong Computer Vision and Pattern Recognition Artificial Intelligence Vision-Language Model (VLM) have gained widespread adoption in Open-Vocabulary (OV) object detection and segmentation tasks. Despite they have shown promise on OV-related tasks, their effectiveness in conventional vision tasks has thus far been unevaluated. In this work, we present the systematic review of VLM-based detection and segmentation, view VLM as the foundational model and conduct comprehensive evaluations across multiple downstream tasks for the first time: 1) The evaluation spans eight detection scenarios (closed-set detection, domain adaptation, crowded objects, etc.) and eight segmentation scenarios (few-shot, open-world, small object, etc.), revealing distinct performance advantages and limitations of various VLM architectures across tasks. 2) As for detection tasks, we evaluate VLMs under three finetuning granularities: \textit{zero prediction}, \textit{visual fine-tuning}, and \textit{text prompt}, and further analyze how different finetuning strategies impact performance under varied task. 3) Based on empirical findings, we provide in-depth analysis of the correlations between task characteristics, model architectures, and training methodologies, offering insights for future VLM design. 4) We believe that this work shall be valuable to the pattern recognition experts working in the fields of computer vision, multimodal learning, and vision foundation models by introducing them to the problem, and familiarizing them with the current status of the progress while providing promising directions for future research. A project associated with this review and evaluation has been created at https://github.com/better-chao/perceptual_abilities_evaluation. |
| title | Vision-Language Model for Object Detection and Segmentation: A Review and Evaluation |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2504.09480 |