GAPS: A Clinically Grounded, Automated Benchmark for Evaluating AI Clinicians
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909965901365248 |
|---|---|
| author | Chen, Xiuyuan Sun, Tao Su, Dexin Yu, Ailing Liu, Junwei Chen, Zhe Jin, Gangzeng Wang, Xin Liu, Jingnan Xiao, Hansong Zhou, Hualei Tao, Dongjie Guo, Chunxiao Yang, Minghui Xia, Yuan Zhao, Jing Fan, Qianrui Wang, Yanyun Zhen, Shuai Chen, Kezhong Wang, Jun Sun, Zewen Zhao, Heng Guan, Tian Wang, Shaodong Chang, Geyun Deng, Jiaming Chen, Hongchengcheng Feng, Kexin Li, Ruzhen Geng, Jiayi Zhao, Changtai Wang, Jun Lin, Guihu Li, Peihao Liu, Liqi Wei, Peng Wang, Jian Gu, Jinjie Wang, Ping Yang, Fan |
| author_facet | Chen, Xiuyuan Sun, Tao Su, Dexin Yu, Ailing Liu, Junwei Chen, Zhe Jin, Gangzeng Wang, Xin Liu, Jingnan Xiao, Hansong Zhou, Hualei Tao, Dongjie Guo, Chunxiao Yang, Minghui Xia, Yuan Zhao, Jing Fan, Qianrui Wang, Yanyun Zhen, Shuai Chen, Kezhong Wang, Jun Sun, Zewen Zhao, Heng Guan, Tian Wang, Shaodong Chang, Geyun Deng, Jiaming Chen, Hongchengcheng Feng, Kexin Li, Ruzhen Geng, Jiayi Zhao, Changtai Wang, Jun Lin, Guihu Li, Peihao Liu, Liqi Wei, Peng Wang, Jian Gu, Jinjie Wang, Ping Yang, Fan |
| contents | Current benchmarks for AI clinician systems, often based on multiple-choice exams or manual rubrics, fail to capture the depth, robustness, and safety required for real-world clinical practice. To address this, we introduce the GAPS framework, a multidimensional paradigm for evaluating Grounding (cognitive depth), Adequacy (answer completeness), Perturbation (robustness), and Safety. Critically, we developed a fully automated, guideline-anchored pipeline to construct a GAPS-aligned benchmark end-to-end, overcoming the scalability and subjectivity limitations of prior work. Our pipeline assembles an evidence neighborhood, creates dual graph and tree representations, and automatically generates questions across G-levels. Rubrics are synthesized by a DeepResearch agent that mimics GRADE-consistent, PICO-driven evidence review in a ReAct loop. Scoring is performed by an ensemble of large language model (LLM) judges. Validation confirmed our automated questions are high-quality and align with clinician judgment (90% agreement, Cohen's Kappa 0.77). Evaluating state-of-the-art models on the benchmark revealed key failure modes: performance degrades sharply with increased reasoning depth (G-axis), models struggle with answer completeness (A-axis), and they are highly vulnerable to adversarial perturbations (P-axis) as well as certain safety issues (S-axis). This automated, clinically-grounded approach provides a reproducible and scalable method for rigorously evaluating AI clinician systems and guiding their development toward safer, more reliable clinical practice. The benchmark dataset GAPS-NSCLC-preview and evaluation code are publicly available at https://huggingface.co/datasets/AQ-MedAI/GAPS-NSCLC-preview and https://github.com/AQ-MedAI/MedicalAiBenchEval. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_13734 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | GAPS: A Clinically Grounded, Automated Benchmark for Evaluating AI Clinicians Chen, Xiuyuan Sun, Tao Su, Dexin Yu, Ailing Liu, Junwei Chen, Zhe Jin, Gangzeng Wang, Xin Liu, Jingnan Xiao, Hansong Zhou, Hualei Tao, Dongjie Guo, Chunxiao Yang, Minghui Xia, Yuan Zhao, Jing Fan, Qianrui Wang, Yanyun Zhen, Shuai Chen, Kezhong Wang, Jun Sun, Zewen Zhao, Heng Guan, Tian Wang, Shaodong Chang, Geyun Deng, Jiaming Chen, Hongchengcheng Feng, Kexin Li, Ruzhen Geng, Jiayi Zhao, Changtai Wang, Jun Lin, Guihu Li, Peihao Liu, Liqi Wei, Peng Wang, Jian Gu, Jinjie Wang, Ping Yang, Fan Computation and Language Current benchmarks for AI clinician systems, often based on multiple-choice exams or manual rubrics, fail to capture the depth, robustness, and safety required for real-world clinical practice. To address this, we introduce the GAPS framework, a multidimensional paradigm for evaluating Grounding (cognitive depth), Adequacy (answer completeness), Perturbation (robustness), and Safety. Critically, we developed a fully automated, guideline-anchored pipeline to construct a GAPS-aligned benchmark end-to-end, overcoming the scalability and subjectivity limitations of prior work. Our pipeline assembles an evidence neighborhood, creates dual graph and tree representations, and automatically generates questions across G-levels. Rubrics are synthesized by a DeepResearch agent that mimics GRADE-consistent, PICO-driven evidence review in a ReAct loop. Scoring is performed by an ensemble of large language model (LLM) judges. Validation confirmed our automated questions are high-quality and align with clinician judgment (90% agreement, Cohen's Kappa 0.77). Evaluating state-of-the-art models on the benchmark revealed key failure modes: performance degrades sharply with increased reasoning depth (G-axis), models struggle with answer completeness (A-axis), and they are highly vulnerable to adversarial perturbations (P-axis) as well as certain safety issues (S-axis). This automated, clinically-grounded approach provides a reproducible and scalable method for rigorously evaluating AI clinician systems and guiding their development toward safer, more reliable clinical practice. The benchmark dataset GAPS-NSCLC-preview and evaluation code are publicly available at https://huggingface.co/datasets/AQ-MedAI/GAPS-NSCLC-preview and https://github.com/AQ-MedAI/MedicalAiBenchEval. |
| title | GAPS: A Clinically Grounded, Automated Benchmark for Evaluating AI Clinicians |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2510.13734 |