ClinConsensus: A Physician-Calibrated Benchmark for Evaluating Clinical Rubric Coverage in Chinese Medical LLMs
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914606255964160 |
|---|---|
| author | Zheng, Xiang Li, Han Luo, Wenjie Zhai, Weiqi Li, Yiyuan Yan, Chuanmiao Yang, Xue Wu, Kailuan Xu, Ruyi Lu, Tianyun Tang, Tianyi Ma, Yubo Yang, Kexin Liu, Dayiheng Yang, Sen Qu, Lin Zhao, Bing Wei, Hu |
| author_facet | Zheng, Xiang Li, Han Luo, Wenjie Zhai, Weiqi Li, Yiyuan Yan, Chuanmiao Yang, Xue Wu, Kailuan Xu, Ruyi Lu, Tianyun Tang, Tianyi Ma, Yubo Yang, Kexin Liu, Dayiheng Yang, Sen Qu, Lin Zhao, Bing Wei, Hu |
| contents | Open-ended medical LLM evaluation remains weakly grounded in physician-calibrated coverage of clinically relevant response criteria, especially in localized clinical settings. We introduce \textsc{ClinConsensus}, a Chinese medical benchmark of 2{,}500 expert-curated cases spanning 36 specialties, 12 task themes, multiple difficulty levels, and lay-facing versus professional-facing settings. Each case is paired with 30 case-specific binary rubric criteria. To evaluate whether responses satisfy enough physician-authored criteria, we propose \emph{Clinician-Anchored Coverage Score} (CACS), a physician-calibrated threshold metric instantiated at \(k=10\), and develop a dual-judge framework combining a GPT-5.1 grader with a physician-supervised Qwen3-8B judge. Evaluating 11 frontier LLMs, we find a persistent coverage gap: Rubric Accuracy ranges from 39.6\% to 52.1\%, whereas CACS@10 ranges from 17.8\% to 32.9\%, leaving a 19.2--21.9 point gap across models. Stratified analyses further reveal substantial variation across reasoning, evidence use, structured extraction, medication instructions, follow-up, and dialogue register. These results suggest that medical LLM evaluation should measure thresholded, rubric-grounded clinical coverage rather than average partial correctness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_02097 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | ClinConsensus: A Physician-Calibrated Benchmark for Evaluating Clinical Rubric Coverage in Chinese Medical LLMs Zheng, Xiang Li, Han Luo, Wenjie Zhai, Weiqi Li, Yiyuan Yan, Chuanmiao Yang, Xue Wu, Kailuan Xu, Ruyi Lu, Tianyun Tang, Tianyi Ma, Yubo Yang, Kexin Liu, Dayiheng Yang, Sen Qu, Lin Zhao, Bing Wei, Hu Computation and Language Open-ended medical LLM evaluation remains weakly grounded in physician-calibrated coverage of clinically relevant response criteria, especially in localized clinical settings. We introduce \textsc{ClinConsensus}, a Chinese medical benchmark of 2{,}500 expert-curated cases spanning 36 specialties, 12 task themes, multiple difficulty levels, and lay-facing versus professional-facing settings. Each case is paired with 30 case-specific binary rubric criteria. To evaluate whether responses satisfy enough physician-authored criteria, we propose \emph{Clinician-Anchored Coverage Score} (CACS), a physician-calibrated threshold metric instantiated at \(k=10\), and develop a dual-judge framework combining a GPT-5.1 grader with a physician-supervised Qwen3-8B judge. Evaluating 11 frontier LLMs, we find a persistent coverage gap: Rubric Accuracy ranges from 39.6\% to 52.1\%, whereas CACS@10 ranges from 17.8\% to 32.9\%, leaving a 19.2--21.9 point gap across models. Stratified analyses further reveal substantial variation across reasoning, evidence use, structured extraction, medication instructions, follow-up, and dialogue register. These results suggest that medical LLM evaluation should measure thresholded, rubric-grounded clinical coverage rather than average partial correctness. |
| title | ClinConsensus: A Physician-Calibrated Benchmark for Evaluating Clinical Rubric Coverage in Chinese Medical LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2603.02097 |