WorldMedQA-V: a multilingual, multimodal medical examination dataset for multimodal language models evaluation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914974666850304 |
|---|---|
| author | Matos, João Chen, Shan Placino, Siena Li, Yingya Pardo, Juan Carlos Climent Idan, Daphna Tohyama, Takeshi Restrepo, David Nakayama, Luis F. Pascual-Leone, Jose M. M. Savova, Guergana Aerts, Hugo Celi, Leo A. Wong, A. Ian Bitterman, Danielle S. Gallifant, Jack |
| author_facet | Matos, João Chen, Shan Placino, Siena Li, Yingya Pardo, Juan Carlos Climent Idan, Daphna Tohyama, Takeshi Restrepo, David Nakayama, Luis F. Pascual-Leone, Jose M. M. Savova, Guergana Aerts, Hugo Celi, Leo A. Wong, A. Ian Bitterman, Danielle S. Gallifant, Jack |
| contents | Multimodal/vision language models (VLMs) are increasingly being deployed in healthcare settings worldwide, necessitating robust benchmarks to ensure their safety, efficacy, and fairness. Multiple-choice question and answer (QA) datasets derived from national medical examinations have long served as valuable evaluation tools, but existing datasets are largely text-only and available in a limited subset of languages and countries. To address these challenges, we present WorldMedQA-V, an updated multilingual, multimodal benchmarking dataset designed to evaluate VLMs in healthcare. WorldMedQA-V includes 568 labeled multiple-choice QAs paired with 568 medical images from four countries (Brazil, Israel, Japan, and Spain), covering original languages and validated English translations by native clinicians, respectively. Baseline performance for common open- and closed-source models are provided in the local language and English translations, and with and without images provided to the model. The WorldMedQA-V benchmark aims to better match AI systems to the diverse healthcare environments in which they are deployed, fostering more equitable, effective, and representative applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_12722 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | WorldMedQA-V: a multilingual, multimodal medical examination dataset for multimodal language models evaluation Matos, João Chen, Shan Placino, Siena Li, Yingya Pardo, Juan Carlos Climent Idan, Daphna Tohyama, Takeshi Restrepo, David Nakayama, Luis F. Pascual-Leone, Jose M. M. Savova, Guergana Aerts, Hugo Celi, Leo A. Wong, A. Ian Bitterman, Danielle S. Gallifant, Jack Computation and Language Multimodal/vision language models (VLMs) are increasingly being deployed in healthcare settings worldwide, necessitating robust benchmarks to ensure their safety, efficacy, and fairness. Multiple-choice question and answer (QA) datasets derived from national medical examinations have long served as valuable evaluation tools, but existing datasets are largely text-only and available in a limited subset of languages and countries. To address these challenges, we present WorldMedQA-V, an updated multilingual, multimodal benchmarking dataset designed to evaluate VLMs in healthcare. WorldMedQA-V includes 568 labeled multiple-choice QAs paired with 568 medical images from four countries (Brazil, Israel, Japan, and Spain), covering original languages and validated English translations by native clinicians, respectively. Baseline performance for common open- and closed-source models are provided in the local language and English translations, and with and without images provided to the model. The WorldMedQA-V benchmark aims to better match AI systems to the diverse healthcare environments in which they are deployed, fostering more equitable, effective, and representative applications. |
| title | WorldMedQA-V: a multilingual, multimodal medical examination dataset for multimodal language models evaluation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.12722 |