Human-Level and Beyond: Benchmarking Large Language Models Against Clinical Pharmacists in Prescription Review
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911296805404672 |
|---|---|
| author | Yang, Yan Bian, Mouxiao Li, Peiling Wen, Bingjian Chen, Ruiyao Mao, Kangkun Ye, Xiaojun Li, Tianbin Chen, Pengcheng Han, Bing Xu, Jie Qiu, Kaifeng Wu, Junyan |
| author_facet | Yang, Yan Bian, Mouxiao Li, Peiling Wen, Bingjian Chen, Ruiyao Mao, Kangkun Ye, Xiaojun Li, Tianbin Chen, Pengcheng Han, Bing Xu, Jie Qiu, Kaifeng Wu, Junyan |
| contents | The rapid advancement of large language models (LLMs) has accelerated their integration into clinical decision support, particularly in prescription review. To enable systematic and fine-grained evaluation, we developed RxBench, a comprehensive benchmark that covers common prescription review categories and consolidates 14 frequent types of prescription errors drawn from authoritative pharmacy references. RxBench consists of 1,150 single-choice, 230 multiple-choice, and 879 short-answer items, all reviewed by experienced clinical pharmacists. We benchmarked 18 state-of-the-art LLMs and identified clear stratification of performance across tasks. Notably, Gemini-2.5-pro-preview-05-06, Grok-4-0709, and DeepSeek-R1-0528 consistently formed the first tier, outperforming other models in both accuracy and robustness. Comparisons with licensed pharmacists indicated that leading LLMs can match or exceed human performance in certain tasks. Furthermore, building on insights from our benchmark evaluation, we performed targeted fine-tuning on a mid-tier model, resulting in a specialized model that rivals leading general-purpose LLMs in performance on short-answer question tasks. The main contribution of RxBench lies in establishing a standardized, error-type-oriented framework that not only reveals the capabilities and limitations of frontier LLMs in prescription review but also provides a foundational resource for building more reliable and specialized clinical tools. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_02024 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Human-Level and Beyond: Benchmarking Large Language Models Against Clinical Pharmacists in Prescription Review Yang, Yan Bian, Mouxiao Li, Peiling Wen, Bingjian Chen, Ruiyao Mao, Kangkun Ye, Xiaojun Li, Tianbin Chen, Pengcheng Han, Bing Xu, Jie Qiu, Kaifeng Wu, Junyan Computation and Language Computers and Society The rapid advancement of large language models (LLMs) has accelerated their integration into clinical decision support, particularly in prescription review. To enable systematic and fine-grained evaluation, we developed RxBench, a comprehensive benchmark that covers common prescription review categories and consolidates 14 frequent types of prescription errors drawn from authoritative pharmacy references. RxBench consists of 1,150 single-choice, 230 multiple-choice, and 879 short-answer items, all reviewed by experienced clinical pharmacists. We benchmarked 18 state-of-the-art LLMs and identified clear stratification of performance across tasks. Notably, Gemini-2.5-pro-preview-05-06, Grok-4-0709, and DeepSeek-R1-0528 consistently formed the first tier, outperforming other models in both accuracy and robustness. Comparisons with licensed pharmacists indicated that leading LLMs can match or exceed human performance in certain tasks. Furthermore, building on insights from our benchmark evaluation, we performed targeted fine-tuning on a mid-tier model, resulting in a specialized model that rivals leading general-purpose LLMs in performance on short-answer question tasks. The main contribution of RxBench lies in establishing a standardized, error-type-oriented framework that not only reveals the capabilities and limitations of frontier LLMs in prescription review but also provides a foundational resource for building more reliable and specialized clinical tools. |
| title | Human-Level and Beyond: Benchmarking Large Language Models Against Clinical Pharmacists in Prescription Review |
| topic | Computation and Language Computers and Society |
| url | https://arxiv.org/abs/2512.02024 |