Describe Anything in Medical Images
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908377827770368 |
|---|---|
| author | Xiao, Xi Zhang, Yunbei Nguyen, Thanh-Huy Lam, Ba-Thinh Wang, Janet Zhao, Lin Hamm, Jihun Wang, Tianyang Li, Xingjian Wang, Xiao Xu, Hao Liu, Tianming Xu, Min |
| author_facet | Xiao, Xi Zhang, Yunbei Nguyen, Thanh-Huy Lam, Ba-Thinh Wang, Janet Zhao, Lin Hamm, Jihun Wang, Tianyang Li, Xingjian Wang, Xiao Xu, Hao Liu, Tianming Xu, Min |
| contents | Localized image captioning has made significant progress with models like the Describe Anything Model (DAM), which can generate detailed region-specific descriptions without explicit region-text supervision. However, such capabilities have yet to be widely applied to specialized domains like medical imaging, where diagnostic interpretation relies on subtle regional findings rather than global understanding. To mitigate this gap, we propose MedDAM, the first comprehensive framework leveraging large vision-language models for region-specific captioning in medical images. MedDAM employs medical expert-designed prompts tailored to specific imaging modalities and establishes a robust evaluation benchmark comprising a customized assessment protocol, data pre-processing pipeline, and specialized QA template library. This benchmark evaluates both MedDAM and other adaptable large vision-language models, focusing on clinical factuality through attribute-level verification tasks, thereby circumventing the absence of ground-truth region-caption pairs in medical datasets. Extensive experiments on the VinDr-CXR, LIDC-IDRI, and SkinCon datasets demonstrate MedDAM's superiority over leading peers (including GPT-4o, Claude 3.7 Sonnet, LLaMA-3.2 Vision, Qwen2.5-VL, GPT-4Rol, and OMG-LLaVA) in the task, revealing the importance of region-level semantic alignment in medical image understanding and establishing MedDAM as a promising foundation for clinical vision-language integration. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_05804 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Describe Anything in Medical Images Xiao, Xi Zhang, Yunbei Nguyen, Thanh-Huy Lam, Ba-Thinh Wang, Janet Zhao, Lin Hamm, Jihun Wang, Tianyang Li, Xingjian Wang, Xiao Xu, Hao Liu, Tianming Xu, Min Computer Vision and Pattern Recognition Localized image captioning has made significant progress with models like the Describe Anything Model (DAM), which can generate detailed region-specific descriptions without explicit region-text supervision. However, such capabilities have yet to be widely applied to specialized domains like medical imaging, where diagnostic interpretation relies on subtle regional findings rather than global understanding. To mitigate this gap, we propose MedDAM, the first comprehensive framework leveraging large vision-language models for region-specific captioning in medical images. MedDAM employs medical expert-designed prompts tailored to specific imaging modalities and establishes a robust evaluation benchmark comprising a customized assessment protocol, data pre-processing pipeline, and specialized QA template library. This benchmark evaluates both MedDAM and other adaptable large vision-language models, focusing on clinical factuality through attribute-level verification tasks, thereby circumventing the absence of ground-truth region-caption pairs in medical datasets. Extensive experiments on the VinDr-CXR, LIDC-IDRI, and SkinCon datasets demonstrate MedDAM's superiority over leading peers (including GPT-4o, Claude 3.7 Sonnet, LLaMA-3.2 Vision, Qwen2.5-VL, GPT-4Rol, and OMG-LLaVA) in the task, revealing the importance of region-level semantic alignment in medical image understanding and establishing MedDAM as a promising foundation for clinical vision-language integration. |
| title | Describe Anything in Medical Images |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2505.05804 |