Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2503.17599 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910243624058880 |
|---|---|
| author | Li, Zheqing Yang, Yiying Lang, Jiping Jiang, Wenhao Chen, Junrong Zhao, Yuhang Li, Shuang Wang, Dingqian Lin, Zhu Li, Xuanna Tang, Yuze Qiu, Jiexian Lu, Xiaolin Yu, Hongji Chen, Shuang Bi, Yuhua Zeng, Xiaofei Chen, Yixian Yao, Lin |
| author_facet | Li, Zheqing Yang, Yiying Lang, Jiping Jiang, Wenhao Chen, Junrong Zhao, Yuhang Li, Shuang Wang, Dingqian Lin, Zhu Li, Xuanna Tang, Yuze Qiu, Jiexian Lu, Xiaolin Yu, Hongji Chen, Shuang Bi, Yuhua Zeng, Xiaofei Chen, Yixian Yao, Lin |
| contents | Large Language Models (LLMs) have demonstrated considerable potential in general practice. However, existing benchmarks and evaluation frameworks primarily depend on exam-style or simplified question-answer formats, lacking a competency-based structure aligned with the real-world clinical responsibilities encountered in general practice. Consequently, the extent to which LLMs can reliably fulfill the duties of general practitioners (GPs) remains uncertain. In this work, we propose a novel evaluation framework to assess the capability of LLMs to function as GPs. Based on this framework, we introduce a general practice benchmark (GPBench), whose data are meticulously annotated by domain experts in accordance with routine clinical practice standards. We evaluate ten state-of-the-art LLMs and analyze their competencies. Our findings indicate that current LLMs are not suitable for autonomous deployment in clinical general practice and that all realistic applications require continuous human oversight; further optimization specifically tailored to the daily responsibilities of GPs remains essential. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_17599 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Evaluating Clinical Competencies of Large Language Models with a General Practice Benchmark Li, Zheqing Yang, Yiying Lang, Jiping Jiang, Wenhao Chen, Junrong Zhao, Yuhang Li, Shuang Wang, Dingqian Lin, Zhu Li, Xuanna Tang, Yuze Qiu, Jiexian Lu, Xiaolin Yu, Hongji Chen, Shuang Bi, Yuhua Zeng, Xiaofei Chen, Yixian Yao, Lin Computation and Language Artificial Intelligence Large Language Models (LLMs) have demonstrated considerable potential in general practice. However, existing benchmarks and evaluation frameworks primarily depend on exam-style or simplified question-answer formats, lacking a competency-based structure aligned with the real-world clinical responsibilities encountered in general practice. Consequently, the extent to which LLMs can reliably fulfill the duties of general practitioners (GPs) remains uncertain. In this work, we propose a novel evaluation framework to assess the capability of LLMs to function as GPs. Based on this framework, we introduce a general practice benchmark (GPBench), whose data are meticulously annotated by domain experts in accordance with routine clinical practice standards. We evaluate ten state-of-the-art LLMs and analyze their competencies. Our findings indicate that current LLMs are not suitable for autonomous deployment in clinical general practice and that all realistic applications require continuous human oversight; further optimization specifically tailored to the daily responsibilities of GPs remains essential. |
| title | Evaluating Clinical Competencies of Large Language Models with a General Practice Benchmark |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2503.17599 |