DrugRAG: Enhancing Pharmacy LLM Performance Through A Novel Retrieval-Augmented Generation Pipeline
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917516057509888 |
|---|---|
| author | Kazemzadeh, Houman Dizaji, Kiarash Mokhtari Tavakoli, Seyed Reza Davoodi, Farbod KarimiNejad, MohammadReza Azad, Parham Abed Latifi, Fatemeh Sabzi, Ali Khosravi, Armin Ahmadi, Siavash Khalaj, Babak Rohban, Mohammad Hossein Aminian, Glolamali Amoozgar, Zohreh Javaheri, Tahereh |
| author_facet | Kazemzadeh, Houman Dizaji, Kiarash Mokhtari Tavakoli, Seyed Reza Davoodi, Farbod KarimiNejad, MohammadReza Azad, Parham Abed Latifi, Fatemeh Sabzi, Ali Khosravi, Armin Ahmadi, Siavash Khalaj, Babak Rohban, Mohammad Hossein Aminian, Glolamali Amoozgar, Zohreh Javaheri, Tahereh |
| contents | In our study, we evaluated large language model (LLM) performance on pharmacy licensure-style question-answering tasks and developed an external knowledge integration method to improve accuracy. We benchmarked ten LLMs with varying parameter sizes (8 billion to 70+ billion) using a 141-question pharmacy dataset, measuring baseline accuracy without modification. Baseline performance ranged from 46% to 92%, with GPT-5 (92%) and o3 (89%) achieving the highest scores, while smaller open-source models showed substantially lower performance. We then developed DrugRAG, a three-step retrieval-augmented generation (RAG) pipeline that retrieves structured, evidence-based drug information and augments model prompts with contextual pharmacological evidence, operating externally and requiring no changes to model architecture or parameters. DrugRAG increased accuracy across all five evaluated models, with gains ranging from 7 to 21 percentage points (e.g., Gemma 3 27B: 61.0% to 71%, Llama 3.1 8B: 46% to 67%). McNemar analyses demonstrated statistically significant paired improvements primarily in smaller and mid-sized open-source models. These findings demonstrate that integrating structured external drug knowledge via DrugRAG can improve LLM performance on pharmacy-focused question-answering tasks without modifying the underlying models, providing a practical pipeline for enhancing evidence-based pharmacy-focused AI applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_14896 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DrugRAG: Enhancing Pharmacy LLM Performance Through A Novel Retrieval-Augmented Generation Pipeline Kazemzadeh, Houman Dizaji, Kiarash Mokhtari Tavakoli, Seyed Reza Davoodi, Farbod KarimiNejad, MohammadReza Azad, Parham Abed Latifi, Fatemeh Sabzi, Ali Khosravi, Armin Ahmadi, Siavash Khalaj, Babak Rohban, Mohammad Hossein Aminian, Glolamali Amoozgar, Zohreh Javaheri, Tahereh Computation and Language Artificial Intelligence In our study, we evaluated large language model (LLM) performance on pharmacy licensure-style question-answering tasks and developed an external knowledge integration method to improve accuracy. We benchmarked ten LLMs with varying parameter sizes (8 billion to 70+ billion) using a 141-question pharmacy dataset, measuring baseline accuracy without modification. Baseline performance ranged from 46% to 92%, with GPT-5 (92%) and o3 (89%) achieving the highest scores, while smaller open-source models showed substantially lower performance. We then developed DrugRAG, a three-step retrieval-augmented generation (RAG) pipeline that retrieves structured, evidence-based drug information and augments model prompts with contextual pharmacological evidence, operating externally and requiring no changes to model architecture or parameters. DrugRAG increased accuracy across all five evaluated models, with gains ranging from 7 to 21 percentage points (e.g., Gemma 3 27B: 61.0% to 71%, Llama 3.1 8B: 46% to 67%). McNemar analyses demonstrated statistically significant paired improvements primarily in smaller and mid-sized open-source models. These findings demonstrate that integrating structured external drug knowledge via DrugRAG can improve LLM performance on pharmacy-focused question-answering tasks without modifying the underlying models, providing a practical pipeline for enhancing evidence-based pharmacy-focused AI applications. |
| title | DrugRAG: Enhancing Pharmacy LLM Performance Through A Novel Retrieval-Augmented Generation Pipeline |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2512.14896 |