Benchmarking Vision-Language Contrastive Methods for Medical Representation Learning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910482291490816 |
|---|---|
| author | Roy, Shuvendu Parhizkar, Yasaman Ogidi, Franklin Khazaie, Vahid Reza Colacci, Michael Etemad, Ali Dolatabadi, Elham Afkanpour, Arash |
| author_facet | Roy, Shuvendu Parhizkar, Yasaman Ogidi, Franklin Khazaie, Vahid Reza Colacci, Michael Etemad, Ali Dolatabadi, Elham Afkanpour, Arash |
| contents | We perform a comprehensive benchmarking of contrastive frameworks for learning multimodal representations in the medical domain. Through this study, we aim to answer the following research questions: (i) How transferable are general-domain representations to the medical domain? (ii) Is multimodal contrastive training sufficient, or does it benefit from unimodal training as well? (iii) What is the impact of feature granularity on the effectiveness of multimodal medical representation learning? To answer these questions, we investigate eight contrastive learning approaches under identical training setups, and train them on 2.8 million image-text pairs from four datasets, and evaluate them on 25 downstream tasks, including classification (zero-shot and linear probing), image-to-text and text-to-image retrieval, and visual question-answering. Our findings suggest a positive answer to the first question, a negative answer to the second question, and the benefit of learning fine-grained features. Finally, we make our code publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_07450 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Benchmarking Vision-Language Contrastive Methods for Medical Representation Learning Roy, Shuvendu Parhizkar, Yasaman Ogidi, Franklin Khazaie, Vahid Reza Colacci, Michael Etemad, Ali Dolatabadi, Elham Afkanpour, Arash Computer Vision and Pattern Recognition Machine Learning We perform a comprehensive benchmarking of contrastive frameworks for learning multimodal representations in the medical domain. Through this study, we aim to answer the following research questions: (i) How transferable are general-domain representations to the medical domain? (ii) Is multimodal contrastive training sufficient, or does it benefit from unimodal training as well? (iii) What is the impact of feature granularity on the effectiveness of multimodal medical representation learning? To answer these questions, we investigate eight contrastive learning approaches under identical training setups, and train them on 2.8 million image-text pairs from four datasets, and evaluate them on 25 downstream tasks, including classification (zero-shot and linear probing), image-to-text and text-to-image retrieval, and visual question-answering. Our findings suggest a positive answer to the first question, a negative answer to the second question, and the benefit of learning fine-grained features. Finally, we make our code publicly available. |
| title | Benchmarking Vision-Language Contrastive Methods for Medical Representation Learning |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2406.07450 |