Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916222386307072 |
|---|---|
| author | Ha, Cuong Nhat Asaadi, Shima Karn, Sanjeev Kumar Farri, Oladimeji Heimann, Tobias Runkler, Thomas |
| author_facet | Ha, Cuong Nhat Asaadi, Shima Karn, Sanjeev Kumar Farri, Oladimeji Heimann, Tobias Runkler, Thomas |
| contents | Vision-language models, while effective in general domains and showing strong performance in diverse multi-modal applications like visual question-answering (VQA), struggle to maintain the same level of effectiveness in more specialized domains, e.g., medical. We propose a medical vision-language model that integrates large vision and language models adapted for the medical domain. This model goes through three stages of parameter-efficient training using three separate biomedical and radiology multi-modal visual and text datasets. The proposed model achieves state-of-the-art performance on the SLAKE 1.0 medical VQA (MedVQA) dataset with an overall accuracy of 87.5% and demonstrates strong performance on another MedVQA dataset, VQA-RAD, achieving an overall accuracy of 73.2%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_16192 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering Ha, Cuong Nhat Asaadi, Shima Karn, Sanjeev Kumar Farri, Oladimeji Heimann, Tobias Runkler, Thomas Computation and Language Computer Vision and Pattern Recognition Vision-language models, while effective in general domains and showing strong performance in diverse multi-modal applications like visual question-answering (VQA), struggle to maintain the same level of effectiveness in more specialized domains, e.g., medical. We propose a medical vision-language model that integrates large vision and language models adapted for the medical domain. This model goes through three stages of parameter-efficient training using three separate biomedical and radiology multi-modal visual and text datasets. The proposed model achieves state-of-the-art performance on the SLAKE 1.0 medical VQA (MedVQA) dataset with an overall accuracy of 87.5% and demonstrates strong performance on another MedVQA dataset, VQA-RAD, achieving an overall accuracy of 73.2%. |
| title | Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering |
| topic | Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2404.16192 |