MedBench: A Comprehensive, Standardized, and Reliable Benchmarking System for Evaluating Chinese Medical Large Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917722738130944 |
|---|---|
| author | Liu, Mianxin Ding, Jinru Xu, Jie Hu, Weiguo Li, Xiaoyang Zhu, Lifeng Bai, Zhian Shi, Xiaoming Wang, Benyou Song, Haitao Liu, Pengfei Zhang, Xiaofan Wang, Shanshan Li, Kang Wang, Haofen Ruan, Tong Huang, Xuanjing Sun, Xin Zhang, Shaoting |
| author_facet | Liu, Mianxin Ding, Jinru Xu, Jie Hu, Weiguo Li, Xiaoyang Zhu, Lifeng Bai, Zhian Shi, Xiaoming Wang, Benyou Song, Haitao Liu, Pengfei Zhang, Xiaofan Wang, Shanshan Li, Kang Wang, Haofen Ruan, Tong Huang, Xuanjing Sun, Xin Zhang, Shaoting |
| contents | Ensuring the general efficacy and goodness for human beings from medical large language models (LLM) before real-world deployment is crucial. However, a widely accepted and accessible evaluation process for medical LLM, especially in the Chinese context, remains to be established. In this work, we introduce "MedBench", a comprehensive, standardized, and reliable benchmarking system for Chinese medical LLM. First, MedBench assembles the currently largest evaluation dataset (300,901 questions) to cover 43 clinical specialties and performs multi-facet evaluation on medical LLM. Second, MedBench provides a standardized and fully automatic cloud-based evaluation infrastructure, with physical separations for question and ground truth. Third, MedBench implements dynamic evaluation mechanisms to prevent shortcut learning and answer remembering. Applying MedBench to popular general and medical LLMs, we observe unbiased, reproducible evaluation results largely aligning with medical professionals' perspectives. This study establishes a significant foundation for preparing the practical applications of Chinese medical LLMs. MedBench is publicly accessible at https://medbench.opencompass.org.cn. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_10990 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MedBench: A Comprehensive, Standardized, and Reliable Benchmarking System for Evaluating Chinese Medical Large Language Models Liu, Mianxin Ding, Jinru Xu, Jie Hu, Weiguo Li, Xiaoyang Zhu, Lifeng Bai, Zhian Shi, Xiaoming Wang, Benyou Song, Haitao Liu, Pengfei Zhang, Xiaofan Wang, Shanshan Li, Kang Wang, Haofen Ruan, Tong Huang, Xuanjing Sun, Xin Zhang, Shaoting Computation and Language Artificial Intelligence Ensuring the general efficacy and goodness for human beings from medical large language models (LLM) before real-world deployment is crucial. However, a widely accepted and accessible evaluation process for medical LLM, especially in the Chinese context, remains to be established. In this work, we introduce "MedBench", a comprehensive, standardized, and reliable benchmarking system for Chinese medical LLM. First, MedBench assembles the currently largest evaluation dataset (300,901 questions) to cover 43 clinical specialties and performs multi-facet evaluation on medical LLM. Second, MedBench provides a standardized and fully automatic cloud-based evaluation infrastructure, with physical separations for question and ground truth. Third, MedBench implements dynamic evaluation mechanisms to prevent shortcut learning and answer remembering. Applying MedBench to popular general and medical LLMs, we observe unbiased, reproducible evaluation results largely aligning with medical professionals' perspectives. This study establishes a significant foundation for preparing the practical applications of Chinese medical LLMs. MedBench is publicly accessible at https://medbench.opencompass.org.cn. |
| title | MedBench: A Comprehensive, Standardized, and Reliable Benchmarking System for Evaluating Chinese Medical Large Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2407.10990 |