RxEval: A Prescription-Level Benchmark for Evaluating LLM Medication Recommendation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909042217058304 |
|---|---|
| author | Chen, Shuhao Jiang, Weisen Wang, Changmiao Wu, Xiaoqing Shi, Xuanren Zhang, Yu Kwok, James T. |
| author_facet | Chen, Shuhao Jiang, Weisen Wang, Changmiao Wu, Xiaoqing Shi, Xuanren Zhang, Yu Kwok, James T. |
| contents | Inpatient medication recommendation requires clinicians to repeatedly select specific medications, doses, and routes as a patient's condition evolves. Existing benchmarks formulate this task as admission-level prediction over coarse drug codes with multi-hot diagnostic and procedure code inputs, failing to capture the per-timepoint, information-rich nature of real prescribing. We propose RxEval, a prescription-level benchmark that evaluates LLM prescribing capability by multiple-choice questions: each question presents a detailed patient profile and time-ordered clinical trajectory, requiring selection of specific medication-dose-route triples from real prescriptions and patient-specific distractors generated via reasoning-chain perturbation. RxEval comprises 1,547 questions spanning 584 patients, 18 diagnostic categories, and 969 unique medications. Evaluation of 16 LLMs shows that RxEval is both challenging and discriminative: F1 ranges from 45.18 to 77.10 across models, and the best Exact Match is only 46.10%. Error analysis reveals that even frontier models may overlook stated patient information and fail to derive clinical conclusions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_14543 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | RxEval: A Prescription-Level Benchmark for Evaluating LLM Medication Recommendation Chen, Shuhao Jiang, Weisen Wang, Changmiao Wu, Xiaoqing Shi, Xuanren Zhang, Yu Kwok, James T. Machine Learning Artificial Intelligence Inpatient medication recommendation requires clinicians to repeatedly select specific medications, doses, and routes as a patient's condition evolves. Existing benchmarks formulate this task as admission-level prediction over coarse drug codes with multi-hot diagnostic and procedure code inputs, failing to capture the per-timepoint, information-rich nature of real prescribing. We propose RxEval, a prescription-level benchmark that evaluates LLM prescribing capability by multiple-choice questions: each question presents a detailed patient profile and time-ordered clinical trajectory, requiring selection of specific medication-dose-route triples from real prescriptions and patient-specific distractors generated via reasoning-chain perturbation. RxEval comprises 1,547 questions spanning 584 patients, 18 diagnostic categories, and 969 unique medications. Evaluation of 16 LLMs shows that RxEval is both challenging and discriminative: F1 ranges from 45.18 to 77.10 across models, and the best Exact Match is only 46.10%. Error analysis reveals that even frontier models may overlook stated patient information and fail to derive clinical conclusions. |
| title | RxEval: A Prescription-Level Benchmark for Evaluating LLM Medication Recommendation |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2605.14543 |