JMed48k: A Multi-Profession Japanese Medical Licensing Benchmark for Vision-Language Model Evaluation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918528756482048 |
|---|---|
| author | Xun, Yue Liu, Junyu Niu, Qian Wang, Xinyi Yuan, Zheng Li, Zirui Zhang, Zequn Zhao, Bowen Wang, Shujun Li, Irene Hatakeyama-Sato, Kan Iwasawa, Yusuke Matsuo, Yutaka |
| author_facet | Xun, Yue Liu, Junyu Niu, Qian Wang, Xinyi Yuan, Zheng Li, Zirui Zhang, Zequn Zhao, Bowen Wang, Shujun Li, Irene Hatakeyama-Sato, Kan Iwasawa, Yusuke Matsuo, Yutaka |
| contents | We introduce JMed48k, a multi-profession Japanese healthcare licensing benchmark for evaluating vision-language models. Built from official PDF materials released by the Japanese Ministry of Health, Labour and Welfare, JMed48k contains 48,862 exam questions and 20,142 images from 11 national licensing examinations between 2005 and 2025, with visual content annotated under an 8-type taxonomy. From this corpus, we derive JMed48k-Eval, a recent five-year evaluation subset with 12,484 scored questions, including 9,905 text-only questions and 2,579 questions with images. We evaluate 21 proprietary, open-source, and medical-specific models, reporting text-only and with-image performance separately. Because these subsets contain different questions, we further introduce a paired image-removal audit that evaluates questions with images before and after removing visual content to explore four answer-transition states. The audit shows that proprietary and open source models gain substantially from images, whereas medical-specific systems show limited observable use of visual evidence, with many correct answers persisting after image removal. Even among proprietary models, the net image-removal effect varies sevenfold across professions, from +5.7 points on Physician questions to +39.8 points on Public Health Nurse questions. We release JMed48k to support reproducible, profession-stratified evaluation of vision-language models in medical licensing settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_22080 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | JMed48k: A Multi-Profession Japanese Medical Licensing Benchmark for Vision-Language Model Evaluation Xun, Yue Liu, Junyu Niu, Qian Wang, Xinyi Yuan, Zheng Li, Zirui Zhang, Zequn Zhao, Bowen Wang, Shujun Li, Irene Hatakeyama-Sato, Kan Iwasawa, Yusuke Matsuo, Yutaka Computer Vision and Pattern Recognition Artificial Intelligence We introduce JMed48k, a multi-profession Japanese healthcare licensing benchmark for evaluating vision-language models. Built from official PDF materials released by the Japanese Ministry of Health, Labour and Welfare, JMed48k contains 48,862 exam questions and 20,142 images from 11 national licensing examinations between 2005 and 2025, with visual content annotated under an 8-type taxonomy. From this corpus, we derive JMed48k-Eval, a recent five-year evaluation subset with 12,484 scored questions, including 9,905 text-only questions and 2,579 questions with images. We evaluate 21 proprietary, open-source, and medical-specific models, reporting text-only and with-image performance separately. Because these subsets contain different questions, we further introduce a paired image-removal audit that evaluates questions with images before and after removing visual content to explore four answer-transition states. The audit shows that proprietary and open source models gain substantially from images, whereas medical-specific systems show limited observable use of visual evidence, with many correct answers persisting after image removal. Even among proprietary models, the net image-removal effect varies sevenfold across professions, from +5.7 points on Physician questions to +39.8 points on Public Health Nurse questions. We release JMed48k to support reproducible, profession-stratified evaluation of vision-language models in medical licensing settings. |
| title | JMed48k: A Multi-Profession Japanese Medical Licensing Benchmark for Vision-Language Model Evaluation |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2605.22080 |