Prompt as Knowledge Bank: Boost Vision-language model via Structural Representation for zero-shot medical detection
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916626006278144 |
|---|---|
| author | Yang, Yuguang Chen, Tongfei Huang, Haoyu Yang, Linlin Xie, Chunyu Leng, Dawei Cao, Xianbin Zhang, Baochang |
| author_facet | Yang, Yuguang Chen, Tongfei Huang, Haoyu Yang, Linlin Xie, Chunyu Leng, Dawei Cao, Xianbin Zhang, Baochang |
| contents | Zero-shot medical detection can further improve detection performance without relying on annotated medical images even upon the fine-tuned model, showing great clinical value. Recent studies leverage grounded vision-language models (GLIP) to achieve this by using detailed disease descriptions as prompts for the target disease name during the inference phase. However, these methods typically treat prompts as equivalent context to the target name, making it difficult to assign specific disease knowledge based on visual information, leading to a coarse alignment between images and target descriptions. In this paper, we propose StructuralGLIP, which introduces an auxiliary branch to encode prompts into a latent knowledge bank layer-by-layer, enabling more context-aware and fine-grained alignment. Specifically, in each layer, we select highly similar features from both the image representation and the knowledge bank, forming structural representations that capture nuanced relationships between image patches and target descriptions. These features are then fused across modalities to further enhance detection performance. Extensive experiments demonstrate that StructuralGLIP achieves a +4.1\% AP improvement over prior state-of-the-art methods across seven zero-shot medical detection benchmarks, and consistently improves fine-tuned models by +3.2\% AP on endoscopy image datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_16223 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Prompt as Knowledge Bank: Boost Vision-language model via Structural Representation for zero-shot medical detection Yang, Yuguang Chen, Tongfei Huang, Haoyu Yang, Linlin Xie, Chunyu Leng, Dawei Cao, Xianbin Zhang, Baochang Computer Vision and Pattern Recognition Zero-shot medical detection can further improve detection performance without relying on annotated medical images even upon the fine-tuned model, showing great clinical value. Recent studies leverage grounded vision-language models (GLIP) to achieve this by using detailed disease descriptions as prompts for the target disease name during the inference phase. However, these methods typically treat prompts as equivalent context to the target name, making it difficult to assign specific disease knowledge based on visual information, leading to a coarse alignment between images and target descriptions. In this paper, we propose StructuralGLIP, which introduces an auxiliary branch to encode prompts into a latent knowledge bank layer-by-layer, enabling more context-aware and fine-grained alignment. Specifically, in each layer, we select highly similar features from both the image representation and the knowledge bank, forming structural representations that capture nuanced relationships between image patches and target descriptions. These features are then fused across modalities to further enhance detection performance. Extensive experiments demonstrate that StructuralGLIP achieves a +4.1\% AP improvement over prior state-of-the-art methods across seven zero-shot medical detection benchmarks, and consistently improves fine-tuned models by +3.2\% AP on endoscopy image datasets. |
| title | Prompt as Knowledge Bank: Boost Vision-language model via Structural Representation for zero-shot medical detection |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2502.16223 |