"Crash Test Dummies" for AI-Enabled Clinical Assessment: Validating Virtual Patient Scenarios with Virtual Learners
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918305084735488 |
|---|---|
| author | Gin, Brian Lim, Ahreum Oliveira, Flávia Silva e Xing, Kuan Song, Xiaomei Amiyangoda, Gayana Seneviratne, Thilanka Doubleday, Alison F. Gangopadhyaya, Ananya Kiser, Bob Shum-Tim, Lukas Patel, Dhruva Marambe, Kosala Maggio, Lauren Tekian, Ara Park, Yoon Soo |
| author_facet | Gin, Brian Lim, Ahreum Oliveira, Flávia Silva e Xing, Kuan Song, Xiaomei Amiyangoda, Gayana Seneviratne, Thilanka Doubleday, Alison F. Gangopadhyaya, Ananya Kiser, Bob Shum-Tim, Lukas Patel, Dhruva Marambe, Kosala Maggio, Lauren Tekian, Ara Park, Yoon Soo |
| contents | Background: In medical and health professions education (HPE), AI is increasingly used to assess clinical competencies, including via virtual standardized patients. However, most evaluations rely on AI-human interrater reliability and lack a measurement framework for how cases, learners, and raters jointly shape scores. This leaves robustness uncertain and can expose learners to misguidance from unvalidated systems. We address this by using AI "simulated learners" to stress-test and psychometrically characterize assessment pipelines before human use.
Objective: Develop an open-source AI virtual patient platform and measurement model for robust competency evaluation across cases and rating conditions.
Methods: We built a platform with virtual patients, virtual learners with tunable ACGME-aligned competency profiles, and multiple independent AI raters scoring encounters with structured Key-Features items. Transcripts were analyzed with a Bayesian HRM-SDT model that treats ratings as decisions under uncertainty and separates learner ability, case performance, and rater behavior; parameters were estimated with MCMC.
Results: The model recovered simulated learners' competencies, with significant correlations to the generating competencies across all ACGME domains despite a non-deterministic pipeline. It estimated case difficulty by competency and showed stable rater detection (sensitivity) and criteria (severity/leniency thresholds) across AI raters using identical models/prompts but different seeds. We also propose a staged "safety blueprint" for deploying AI tools with learners, tied to entrustment-based validation milestones.
Conclusions: Combining a purpose-built virtual patient platform with a principled psychometric model enables robust, interpretable, generalizable competency estimates and supports validation of AI-assisted assessment prior to use with human learners. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_18085 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | "Crash Test Dummies" for AI-Enabled Clinical Assessment: Validating Virtual Patient Scenarios with Virtual Learners Gin, Brian Lim, Ahreum Oliveira, Flávia Silva e Xing, Kuan Song, Xiaomei Amiyangoda, Gayana Seneviratne, Thilanka Doubleday, Alison F. Gangopadhyaya, Ananya Kiser, Bob Shum-Tim, Lukas Patel, Dhruva Marambe, Kosala Maggio, Lauren Tekian, Ara Park, Yoon Soo Human-Computer Interaction Artificial Intelligence Applications Background: In medical and health professions education (HPE), AI is increasingly used to assess clinical competencies, including via virtual standardized patients. However, most evaluations rely on AI-human interrater reliability and lack a measurement framework for how cases, learners, and raters jointly shape scores. This leaves robustness uncertain and can expose learners to misguidance from unvalidated systems. We address this by using AI "simulated learners" to stress-test and psychometrically characterize assessment pipelines before human use. Objective: Develop an open-source AI virtual patient platform and measurement model for robust competency evaluation across cases and rating conditions. Methods: We built a platform with virtual patients, virtual learners with tunable ACGME-aligned competency profiles, and multiple independent AI raters scoring encounters with structured Key-Features items. Transcripts were analyzed with a Bayesian HRM-SDT model that treats ratings as decisions under uncertainty and separates learner ability, case performance, and rater behavior; parameters were estimated with MCMC. Results: The model recovered simulated learners' competencies, with significant correlations to the generating competencies across all ACGME domains despite a non-deterministic pipeline. It estimated case difficulty by competency and showed stable rater detection (sensitivity) and criteria (severity/leniency thresholds) across AI raters using identical models/prompts but different seeds. We also propose a staged "safety blueprint" for deploying AI tools with learners, tied to entrustment-based validation milestones. Conclusions: Combining a purpose-built virtual patient platform with a principled psychometric model enables robust, interpretable, generalizable competency estimates and supports validation of AI-assisted assessment prior to use with human learners. |
| title | "Crash Test Dummies" for AI-Enabled Clinical Assessment: Validating Virtual Patient Scenarios with Virtual Learners |
| topic | Human-Computer Interaction Artificial Intelligence Applications |
| url | https://arxiv.org/abs/2601.18085 |