Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2511.06738 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915608783749120 |
|---|---|
| author | Kim, Hyunjae Sohn, Jiwoong Gilson, Aidan Cochran-Caggiano, Nicholas Applebaum, Serina Jin, Heeju Park, Seihee Park, Yujin Park, Jiyeong Choi, Seoyoung Contreras, Brittany Alexandra Herrera Huang, Thomas Yun, Jaehoon Wei, Ethan F. Jiang, Roy Colucci, Leah Lai, Eric Dave, Amisha Guo, Tuo Singer, Maxwell B. Koo, Yonghoe Adelman, Ron A. Zou, James Taylor, Andrew Cohan, Arman Xu, Hua Chen, Qingyu |
| author_facet | Kim, Hyunjae Sohn, Jiwoong Gilson, Aidan Cochran-Caggiano, Nicholas Applebaum, Serina Jin, Heeju Park, Seihee Park, Yujin Park, Jiyeong Choi, Seoyoung Contreras, Brittany Alexandra Herrera Huang, Thomas Yun, Jaehoon Wei, Ethan F. Jiang, Roy Colucci, Leah Lai, Eric Dave, Amisha Guo, Tuo Singer, Maxwell B. Koo, Yonghoe Adelman, Ron A. Zou, James Taylor, Andrew Cohan, Arman Xu, Hua Chen, Qingyu |
| contents | Large language models (LLMs) are transforming the landscape of medicine, yet two fundamental challenges persist: keeping up with rapidly evolving medical knowledge and providing verifiable, evidence-grounded reasoning. Retrieval-augmented generation (RAG) has been widely adopted to address these limitations by supplementing model outputs with retrieved evidence. However, whether RAG reliably achieves these goals remains unclear. Here, we present the most comprehensive expert evaluation of RAG in medicine to date. Eighteen medical experts contributed a total of 80,502 annotations, assessing 800 model outputs generated by GPT-4o and Llama-3.1-8B across 200 real-world patient and USMLE-style queries. We systematically decomposed the RAG pipeline into three components: (i) evidence retrieval (relevance of retrieved passages), (ii) evidence selection (accuracy of evidence usage), and (iii) response generation (factuality and completeness of outputs). Contrary to expectation, standard RAG often degraded performance: only 22% of top-16 passages were relevant, evidence selection remained weak (precision 41-43%, recall 27-49%), and factuality and completeness dropped by up to 6% and 5%, respectively, compared with non-RAG variants. Retrieval and evidence selection remain key failure points for the model, contributing to the overall performance drop. We further show that simple yet effective strategies, including evidence filtering and query reformulation, substantially mitigate these issues, improving performance on MedMCQA and MedXpertQA by up to 12% and 8.2%, respectively. These findings call for re-examining RAG's role in medicine and highlight the importance of stage-aware evaluation and deliberate system design for reliable medical LLM applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_06738 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Rethinking Retrieval-Augmented Generation for Medicine: A Large-Scale, Systematic Expert Evaluation and Practical Insights Kim, Hyunjae Sohn, Jiwoong Gilson, Aidan Cochran-Caggiano, Nicholas Applebaum, Serina Jin, Heeju Park, Seihee Park, Yujin Park, Jiyeong Choi, Seoyoung Contreras, Brittany Alexandra Herrera Huang, Thomas Yun, Jaehoon Wei, Ethan F. Jiang, Roy Colucci, Leah Lai, Eric Dave, Amisha Guo, Tuo Singer, Maxwell B. Koo, Yonghoe Adelman, Ron A. Zou, James Taylor, Andrew Cohan, Arman Xu, Hua Chen, Qingyu Computation and Language Large language models (LLMs) are transforming the landscape of medicine, yet two fundamental challenges persist: keeping up with rapidly evolving medical knowledge and providing verifiable, evidence-grounded reasoning. Retrieval-augmented generation (RAG) has been widely adopted to address these limitations by supplementing model outputs with retrieved evidence. However, whether RAG reliably achieves these goals remains unclear. Here, we present the most comprehensive expert evaluation of RAG in medicine to date. Eighteen medical experts contributed a total of 80,502 annotations, assessing 800 model outputs generated by GPT-4o and Llama-3.1-8B across 200 real-world patient and USMLE-style queries. We systematically decomposed the RAG pipeline into three components: (i) evidence retrieval (relevance of retrieved passages), (ii) evidence selection (accuracy of evidence usage), and (iii) response generation (factuality and completeness of outputs). Contrary to expectation, standard RAG often degraded performance: only 22% of top-16 passages were relevant, evidence selection remained weak (precision 41-43%, recall 27-49%), and factuality and completeness dropped by up to 6% and 5%, respectively, compared with non-RAG variants. Retrieval and evidence selection remain key failure points for the model, contributing to the overall performance drop. We further show that simple yet effective strategies, including evidence filtering and query reformulation, substantially mitigate these issues, improving performance on MedMCQA and MedXpertQA by up to 12% and 8.2%, respectively. These findings call for re-examining RAG's role in medicine and highlight the importance of stage-aware evaluation and deliberate system design for reliable medical LLM applications. |
| title | Rethinking Retrieval-Augmented Generation for Medicine: A Large-Scale, Systematic Expert Evaluation and Practical Insights |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2511.06738 |