Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.10755 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910216999665664 |
|---|---|
| author | Ning, Junzhi Lin, Jiashi Fang, Yingying Li, Wei Liu, Jiyao Tang, Cheng Ma, Chenglong Tang, Wenhao Li, Tianbin Huang, Ziyan Yang, Guang He, Junjun |
| author_facet | Ning, Junzhi Lin, Jiashi Fang, Yingying Li, Wei Liu, Jiyao Tang, Cheng Ma, Chenglong Tang, Wenhao Li, Tianbin Huang, Ziyan Yang, Guang He, Junjun |
| contents | Multimodal large language models (MLLMs) have advanced clinical tasks for common conditions, but their performance on rare diseases remains largely untested. In rare-disease scenarios, clinicians often lack prior clinical knowledge, forcing them to rely strictly on case-level evidence for clinical judgments. Existing benchmarks predominantly evaluate common-condition, single-image settings, leaving multimodal and multi-image evidence integration under rare-disease data scarcity systematically unevaluated. We introduce MMRareBench, to our knowledge the first rare-disease benchmark jointly evaluating multimodal and multi-image clinical capability across four workflow-aligned tracks: diagnosis, treatment planning, cross-image evidence alignment, and examination suggestion. The benchmark comprises 1,756 question-answer pairs with 7,958 associated medical images curated from PMC case reports, with Orphanet-anchored ontology alignment, track-specific leakage control, evidence-grounded annotations, and a two-level evaluation protocol. A systematic evaluation of 23 MLLMs reveals fragmented capability profiles and universally low treatment-planning performance, with medical-domain models trailing general-purpose MLLMs substantially on multi-image tracks despite competitive diagnostic scores. These patterns are consistent with a capacity dilution effect: medical fine-tuning can narrow the diagnostic gap but may erode the compositional multi-image capability that rare-disease evidence integration demands. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_10755 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MMRareBench: A Rare-Disease Multimodal and Multi-Image Medical Benchmark Ning, Junzhi Lin, Jiashi Fang, Yingying Li, Wei Liu, Jiyao Tang, Cheng Ma, Chenglong Tang, Wenhao Li, Tianbin Huang, Ziyan Yang, Guang He, Junjun Computer Vision and Pattern Recognition Multimodal large language models (MLLMs) have advanced clinical tasks for common conditions, but their performance on rare diseases remains largely untested. In rare-disease scenarios, clinicians often lack prior clinical knowledge, forcing them to rely strictly on case-level evidence for clinical judgments. Existing benchmarks predominantly evaluate common-condition, single-image settings, leaving multimodal and multi-image evidence integration under rare-disease data scarcity systematically unevaluated. We introduce MMRareBench, to our knowledge the first rare-disease benchmark jointly evaluating multimodal and multi-image clinical capability across four workflow-aligned tracks: diagnosis, treatment planning, cross-image evidence alignment, and examination suggestion. The benchmark comprises 1,756 question-answer pairs with 7,958 associated medical images curated from PMC case reports, with Orphanet-anchored ontology alignment, track-specific leakage control, evidence-grounded annotations, and a two-level evaluation protocol. A systematic evaluation of 23 MLLMs reveals fragmented capability profiles and universally low treatment-planning performance, with medical-domain models trailing general-purpose MLLMs substantially on multi-image tracks despite competitive diagnostic scores. These patterns are consistent with a capacity dilution effect: medical fine-tuning can narrow the diagnostic gap but may erode the compositional multi-image capability that rare-disease evidence integration demands. |
| title | MMRareBench: A Rare-Disease Multimodal and Multi-Image Medical Benchmark |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2604.10755 |