Benchmarking Robustness of Contrastive Learning Models for Medical Image-Report Retrieval
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910786620751872 |
|---|---|
| author | Deanda, Demetrio Masupalli, Yuktha Priya Yang, Jeong Lee, Young Cao, Zechun Liang, Gongbo |
| author_facet | Deanda, Demetrio Masupalli, Yuktha Priya Yang, Jeong Lee, Young Cao, Zechun Liang, Gongbo |
| contents | Medical images and reports offer invaluable insights into patient health. The heterogeneity and complexity of these data hinder effective analysis. To bridge this gap, we investigate contrastive learning models for cross-domain retrieval, which associates medical images with their corresponding clinical reports. This study benchmarks the robustness of four state-of-the-art contrastive learning models: CLIP, CXR-RePaiR, MedCLIP, and CXR-CLIP. We introduce an occlusion retrieval task to evaluate model performance under varying levels of image corruption. Our findings reveal that all evaluated models are highly sensitive to out-of-distribution data, as evidenced by the proportional decrease in performance with increasing occlusion levels. While MedCLIP exhibits slightly more robustness, its overall performance remains significantly behind CXR-CLIP and CXR-RePaiR. CLIP, trained on a general-purpose dataset, struggles with medical image-report retrieval, highlighting the importance of domain-specific training data. The evaluation of this work suggests that more effort needs to be spent on improving the robustness of these models. By addressing these limitations, we can develop more reliable cross-domain retrieval models for medical applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_09134 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Benchmarking Robustness of Contrastive Learning Models for Medical Image-Report Retrieval Deanda, Demetrio Masupalli, Yuktha Priya Yang, Jeong Lee, Young Cao, Zechun Liang, Gongbo Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Information Retrieval Machine Learning Medical images and reports offer invaluable insights into patient health. The heterogeneity and complexity of these data hinder effective analysis. To bridge this gap, we investigate contrastive learning models for cross-domain retrieval, which associates medical images with their corresponding clinical reports. This study benchmarks the robustness of four state-of-the-art contrastive learning models: CLIP, CXR-RePaiR, MedCLIP, and CXR-CLIP. We introduce an occlusion retrieval task to evaluate model performance under varying levels of image corruption. Our findings reveal that all evaluated models are highly sensitive to out-of-distribution data, as evidenced by the proportional decrease in performance with increasing occlusion levels. While MedCLIP exhibits slightly more robustness, its overall performance remains significantly behind CXR-CLIP and CXR-RePaiR. CLIP, trained on a general-purpose dataset, struggles with medical image-report retrieval, highlighting the importance of domain-specific training data. The evaluation of this work suggests that more effort needs to be spent on improving the robustness of these models. By addressing these limitations, we can develop more reliable cross-domain retrieval models for medical applications. |
| title | Benchmarking Robustness of Contrastive Learning Models for Medical Image-Report Retrieval |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Information Retrieval Machine Learning |
| url | https://arxiv.org/abs/2501.09134 |