In-Context Adaptation of VLMs for Few-Shot Cell Detection in Optical Microscopy
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909893944934400 |
|---|---|
| author | Ganguly, Shreyan Biswas, Angona Rade, Jaydeep Hasib, Md Hasibul Hasan Masud, Nabila Singla, Nitish Dash, Abhipsa Bhattacharjee, Ushashi Balu, Aditya Sarkar, Anwesha Krishnamurthy, Adarsh Sarkar, Soumik |
| author_facet | Ganguly, Shreyan Biswas, Angona Rade, Jaydeep Hasib, Md Hasibul Hasan Masud, Nabila Singla, Nitish Dash, Abhipsa Bhattacharjee, Ushashi Balu, Aditya Sarkar, Anwesha Krishnamurthy, Adarsh Sarkar, Soumik |
| contents | Foundation vision-language models (VLMs) excel on natural images, but their utility for biomedical microscopy remains underexplored. In this paper, we investigate how in-context learning enables state-of-the-art VLMs to perform few-shot object detection when large annotated datasets are unavailable, as is often the case with microscopic images. We introduce the Micro-OD benchmark, a curated collection of 252 images specifically curated for in-context learning, with bounding-box annotations spanning 11 cell types across four sources, including two in-lab expert-annotated sets. We systematically evaluate eight VLMs under few-shot conditions and compare variants with and without implicit test-time reasoning tokens. We further implement a hybrid Few-Shot Object Detection (FSOD) pipeline that combines a detection head with a VLM-based few-shot classifier, which enhances the few-shot performance of recent VLMs on our benchmark. Across datasets, we observe that zero-shot performance is weak due to the domain gap; however, few-shot support consistently improves detection, with marginal gains achieved after six shots. We observe that models with reasoning tokens are more effective for end-to-end localization, whereas simpler variants are more suitable for classifying pre-localized crops. Our results highlight in-context adaptation as a practical path for microscopy, and our benchmark provides a reproducible testbed for advancing open-vocabulary detection in biomedical imaging. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_05565 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | In-Context Adaptation of VLMs for Few-Shot Cell Detection in Optical Microscopy Ganguly, Shreyan Biswas, Angona Rade, Jaydeep Hasib, Md Hasibul Hasan Masud, Nabila Singla, Nitish Dash, Abhipsa Bhattacharjee, Ushashi Balu, Aditya Sarkar, Anwesha Krishnamurthy, Adarsh Sarkar, Soumik Computer Vision and Pattern Recognition Artificial Intelligence Foundation vision-language models (VLMs) excel on natural images, but their utility for biomedical microscopy remains underexplored. In this paper, we investigate how in-context learning enables state-of-the-art VLMs to perform few-shot object detection when large annotated datasets are unavailable, as is often the case with microscopic images. We introduce the Micro-OD benchmark, a curated collection of 252 images specifically curated for in-context learning, with bounding-box annotations spanning 11 cell types across four sources, including two in-lab expert-annotated sets. We systematically evaluate eight VLMs under few-shot conditions and compare variants with and without implicit test-time reasoning tokens. We further implement a hybrid Few-Shot Object Detection (FSOD) pipeline that combines a detection head with a VLM-based few-shot classifier, which enhances the few-shot performance of recent VLMs on our benchmark. Across datasets, we observe that zero-shot performance is weak due to the domain gap; however, few-shot support consistently improves detection, with marginal gains achieved after six shots. We observe that models with reasoning tokens are more effective for end-to-end localization, whereas simpler variants are more suitable for classifying pre-localized crops. Our results highlight in-context adaptation as a practical path for microscopy, and our benchmark provides a reproducible testbed for advancing open-vocabulary detection in biomedical imaging. |
| title | In-Context Adaptation of VLMs for Few-Shot Cell Detection in Optical Microscopy |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2511.05565 |