When Cases Get Rare: A Retrieval Benchmark for Off-Guideline Clinical Question Answering
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913151208914944 |
|---|---|
| author | Lee, Doeun Zhang, Muge Yu, Yi Manne, Ashish Koesters, Stephen Wen, Frank Buchanan, Brady Villagomez, Lynda Moninuola, Oluwatoba Lim, James Tobin, Kathryn Srisuwananukorn, Andrew Zhang, Ping Kumar, Sachin |
| author_facet | Lee, Doeun Zhang, Muge Yu, Yi Manne, Ashish Koesters, Stephen Wen, Frank Buchanan, Brady Villagomez, Lynda Moninuola, Oluwatoba Lim, James Tobin, Kathryn Srisuwananukorn, Andrew Zhang, Ping Kumar, Sachin |
| contents | Across medical specialties, clinical practice is anchored in evidence-based guidelines that codify best studied diagnostic and treatment pathways. These pathways routinely fall short for the long tail of real-world care not covered by guidelines. Most medical large language models (LLMs), however, are trained to encode common, guideline-focused medical knowledge in their parameters. Current evaluations test models primarily on recalling and reasoning with this memorized content, often in multiple-choice settings. Given the fundamental importance of evidence-based reasoning in medicine, it is neither feasible nor reliable to depend on memorization in practice. To address this gap, we introduce OGCaReBench, a free-form retrieval-focused benchmark aimed at evaluating LLMs at answering clinical questions that require going beyond typical guidelines. Extracted from published medical case reports and validated by medical experts, OGCaReBench contains long-form clinical questions requiring free-text answers, providing a systematic framework for assessing open-ended medical reasoning in rare, case-based scenarios. Our experiments reveal that even the best-performing baseline (GPT-5.2) correctly answers only 56% of our benchmark with specialized models only reaching 42%. Augmenting models with retrieved medical articles improves this performance to up to 82% (using GPT-5.2) highlighting the importance of evidence-grounding for real-world medical reasoning tasks. This work thus establishes a foundation for benchmarking and advancing both general-purpose and medical LLMs to produce reliable answers in challenging clinical contexts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_21807 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | When Cases Get Rare: A Retrieval Benchmark for Off-Guideline Clinical Question Answering Lee, Doeun Zhang, Muge Yu, Yi Manne, Ashish Koesters, Stephen Wen, Frank Buchanan, Brady Villagomez, Lynda Moninuola, Oluwatoba Lim, James Tobin, Kathryn Srisuwananukorn, Andrew Zhang, Ping Kumar, Sachin Computation and Language Across medical specialties, clinical practice is anchored in evidence-based guidelines that codify best studied diagnostic and treatment pathways. These pathways routinely fall short for the long tail of real-world care not covered by guidelines. Most medical large language models (LLMs), however, are trained to encode common, guideline-focused medical knowledge in their parameters. Current evaluations test models primarily on recalling and reasoning with this memorized content, often in multiple-choice settings. Given the fundamental importance of evidence-based reasoning in medicine, it is neither feasible nor reliable to depend on memorization in practice. To address this gap, we introduce OGCaReBench, a free-form retrieval-focused benchmark aimed at evaluating LLMs at answering clinical questions that require going beyond typical guidelines. Extracted from published medical case reports and validated by medical experts, OGCaReBench contains long-form clinical questions requiring free-text answers, providing a systematic framework for assessing open-ended medical reasoning in rare, case-based scenarios. Our experiments reveal that even the best-performing baseline (GPT-5.2) correctly answers only 56% of our benchmark with specialized models only reaching 42%. Augmenting models with retrieved medical articles improves this performance to up to 82% (using GPT-5.2) highlighting the importance of evidence-grounding for real-world medical reasoning tasks. This work thus establishes a foundation for benchmarking and advancing both general-purpose and medical LLMs to produce reliable answers in challenging clinical contexts. |
| title | When Cases Get Rare: A Retrieval Benchmark for Off-Guideline Clinical Question Answering |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2605.21807 |