LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918133329035264 |
|---|---|
| author | Zhang, Ming Shen, Yujiong Li, Zelin Sha, Huayu Hu, Binze Wang, Yuhui Huang, Chenhao Liu, Shichun Tong, Jingqi Jiang, Changhao Chai, Mingxu Xi, Zhiheng Dou, Shihan Gui, Tao Zhang, Qi Huang, Xuanjing |
| author_facet | Zhang, Ming Shen, Yujiong Li, Zelin Sha, Huayu Hu, Binze Wang, Yuhui Huang, Chenhao Liu, Shichun Tong, Jingqi Jiang, Changhao Chai, Mingxu Xi, Zhiheng Dou, Shihan Gui, Tao Zhang, Qi Huang, Xuanjing |
| contents | Evaluating large language models (LLMs) in medicine is crucial because medical applications require high accuracy with little room for error. Current medical benchmarks have three main types: medical exam-based, comprehensive medical, and specialized assessments. However, these benchmarks have limitations in question design (mostly multiple-choice), data sources (often not derived from real clinical scenarios), and evaluation methods (poor assessment of complex reasoning). To address these issues, we present LLMEval-Med, a new benchmark covering five core medical areas, including 2,996 questions created from real-world electronic health records and expert-designed clinical scenarios. We also design an automated evaluation pipeline, incorporating expert-developed checklists into our LLM-as-Judge framework. Furthermore, our methodology validates machine scoring through human-machine agreement analysis, dynamically refining checklists and prompts based on expert feedback to ensure reliability. We evaluate 13 LLMs across three categories (specialized medical models, open-source models, and closed-source models) on LLMEval-Med, providing valuable insights for the safe and effective deployment of LLMs in medical domains. The dataset is released in https://github.com/llmeval/LLMEval-Med. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_04078 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validation Zhang, Ming Shen, Yujiong Li, Zelin Sha, Huayu Hu, Binze Wang, Yuhui Huang, Chenhao Liu, Shichun Tong, Jingqi Jiang, Changhao Chai, Mingxu Xi, Zhiheng Dou, Shihan Gui, Tao Zhang, Qi Huang, Xuanjing Computation and Language Artificial Intelligence Evaluating large language models (LLMs) in medicine is crucial because medical applications require high accuracy with little room for error. Current medical benchmarks have three main types: medical exam-based, comprehensive medical, and specialized assessments. However, these benchmarks have limitations in question design (mostly multiple-choice), data sources (often not derived from real clinical scenarios), and evaluation methods (poor assessment of complex reasoning). To address these issues, we present LLMEval-Med, a new benchmark covering five core medical areas, including 2,996 questions created from real-world electronic health records and expert-designed clinical scenarios. We also design an automated evaluation pipeline, incorporating expert-developed checklists into our LLM-as-Judge framework. Furthermore, our methodology validates machine scoring through human-machine agreement analysis, dynamically refining checklists and prompts based on expert feedback to ensure reliability. We evaluate 13 LLMs across three categories (specialized medical models, open-source models, and closed-source models) on LLMEval-Med, providing valuable insights for the safe and effective deployment of LLMs in medical domains. The dataset is released in https://github.com/llmeval/LLMEval-Med. |
| title | LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validation |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2506.04078 |