Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.11559 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908772266409984 |
|---|---|
| author | AlDin, Zilal Eiz Wu, John Fung, Jeffrey Paul King, Jennifer Watts, Mya ONeill, Lauren Cross, Adam Richard Sun, Jimeng |
| author_facet | AlDin, Zilal Eiz Wu, John Fung, Jeffrey Paul King, Jennifer Watts, Mya ONeill, Lauren Cross, Adam Richard Sun, Jimeng |
| contents | Despite rare diseases affecting 1 in 10 Americans, their differential diagnosis remains challenging. Due to their impressive recall abilities, large language models (LLMs) have been recently explored for differential diagnosis. Existing approaches to evaluating LLM-based rare disease diagnosis suffer from two critical limitations: they rely on idealized clinical case studies that fail to capture real-world clinical complexity, or they use ICD codes as disease labels, which significantly undercounts rare diseases since many lack direct mappings to comprehensive rare disease databases like Orphanet. To address these limitations, we explore MIMIC-RD, a rare disease differential diagnosis benchmark constructed by directly mapping clinical text entities to Orphanet. Our methodology involved an initial LLM-based mining process followed by validation from four medical annotators to confirm identified entities were genuine rare diseases. We evaluated various models on our dataset of 145 patients and found that current state-of-the-art LLMs perform poorly on rare disease differential diagnosis, highlighting the substantial gap between existing capabilities and clinical needs. From our findings, we outline several future steps towards improving differential diagnosis of rare diseases. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_11559 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MIMIC-RD: Can LLMs differentially diagnose rare diseases in real-world clinical settings? AlDin, Zilal Eiz Wu, John Fung, Jeffrey Paul King, Jennifer Watts, Mya ONeill, Lauren Cross, Adam Richard Sun, Jimeng Artificial Intelligence Computation and Language Machine Learning Despite rare diseases affecting 1 in 10 Americans, their differential diagnosis remains challenging. Due to their impressive recall abilities, large language models (LLMs) have been recently explored for differential diagnosis. Existing approaches to evaluating LLM-based rare disease diagnosis suffer from two critical limitations: they rely on idealized clinical case studies that fail to capture real-world clinical complexity, or they use ICD codes as disease labels, which significantly undercounts rare diseases since many lack direct mappings to comprehensive rare disease databases like Orphanet. To address these limitations, we explore MIMIC-RD, a rare disease differential diagnosis benchmark constructed by directly mapping clinical text entities to Orphanet. Our methodology involved an initial LLM-based mining process followed by validation from four medical annotators to confirm identified entities were genuine rare diseases. We evaluated various models on our dataset of 145 patients and found that current state-of-the-art LLMs perform poorly on rare disease differential diagnosis, highlighting the substantial gap between existing capabilities and clinical needs. From our findings, we outline several future steps towards improving differential diagnosis of rare diseases. |
| title | MIMIC-RD: Can LLMs differentially diagnose rare diseases in real-world clinical settings? |
| topic | Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2601.11559 |