Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2409.18924 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911288847761408 |
|---|---|
| author | Yu, Huizi Zhou, Jiayan Li, Lingyao Chen, Shan Gallifant, Jack Shi, Anye Li, Xiang He, Jingxian Hua, Wenyue Jin, Mingyu Chen, Guang Zhou, Yang Li, Zhao Gupte, Trisha Chen, Ming-Li Azizi, Zahra Dou, Qi Yan, Bryan P. Zhang, Yongfeng Xing, Yanqiu Bitterman, Themistocles L. Danielle S. Assimes, Themistocles L. Ma, Xin Lu, Lin Fan, Lizhou |
| author_facet | Yu, Huizi Zhou, Jiayan Li, Lingyao Chen, Shan Gallifant, Jack Shi, Anye Li, Xiang He, Jingxian Hua, Wenyue Jin, Mingyu Chen, Guang Zhou, Yang Li, Zhao Gupte, Trisha Chen, Ming-Li Azizi, Zahra Dou, Qi Yan, Bryan P. Zhang, Yongfeng Xing, Yanqiu Bitterman, Themistocles L. Danielle S. Assimes, Themistocles L. Ma, Xin Lu, Lin Fan, Lizhou |
| contents | Background: Simulated patient systems are important in medical education and research, providing safe, integrative training environments and supporting clinical decision making. Advances in artificial intelligence (AI), especially large language models (LLMs), can enhance simulated patients by replicating medical conditions and doctor patient interactions with high fidelity and at low cost, but effectiveness and trustworthiness remain open challenges. Methods: We developed AIPatient, a simulated patient system powered by LLM based AI agents. The system uses a retrieval augmented generation (RAG) framework with six task specific agents for complex reasoning. To improve realism, it is linked to the AIPatient knowledge graph built from de identified real patient data in the MIMIC III intensive care database. Results: We evaluated electronic health record (EHR) based medical question answering (QA), readability, robustness, stability, and user experience. AIPatient reached 94.15 percent QA accuracy when all six agents were enabled, outperforming versions with partial or no agent integration. The knowledge base achieved an F1 score of 0.89. Readability scores showed a median Flesch Reading Ease of 68.77 and a median Flesch Kincaid Grade of 6.4, indicating accessibility for most medical trainees and clinicians. Robustness and stability were supported by non significant variance in repeated trials (analysis of variance F value 0.61, p greater than 0.1; F value 0.78, p greater than 0.1). A user study with medical students showed that AIPatient provides high fidelity, usability, and educational value, comparable to or better than human simulated patients for history taking. Conclusions: LLM based simulated patient systems can deliver accurate, readable, and reliable medical encounters and show strong potential to transform medical education. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_18924 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Simulated patient systems powered by large language model-based AI agents offer potential for transforming medical education Yu, Huizi Zhou, Jiayan Li, Lingyao Chen, Shan Gallifant, Jack Shi, Anye Li, Xiang He, Jingxian Hua, Wenyue Jin, Mingyu Chen, Guang Zhou, Yang Li, Zhao Gupte, Trisha Chen, Ming-Li Azizi, Zahra Dou, Qi Yan, Bryan P. Zhang, Yongfeng Xing, Yanqiu Bitterman, Themistocles L. Danielle S. Assimes, Themistocles L. Ma, Xin Lu, Lin Fan, Lizhou Computation and Language Artificial Intelligence Background: Simulated patient systems are important in medical education and research, providing safe, integrative training environments and supporting clinical decision making. Advances in artificial intelligence (AI), especially large language models (LLMs), can enhance simulated patients by replicating medical conditions and doctor patient interactions with high fidelity and at low cost, but effectiveness and trustworthiness remain open challenges. Methods: We developed AIPatient, a simulated patient system powered by LLM based AI agents. The system uses a retrieval augmented generation (RAG) framework with six task specific agents for complex reasoning. To improve realism, it is linked to the AIPatient knowledge graph built from de identified real patient data in the MIMIC III intensive care database. Results: We evaluated electronic health record (EHR) based medical question answering (QA), readability, robustness, stability, and user experience. AIPatient reached 94.15 percent QA accuracy when all six agents were enabled, outperforming versions with partial or no agent integration. The knowledge base achieved an F1 score of 0.89. Readability scores showed a median Flesch Reading Ease of 68.77 and a median Flesch Kincaid Grade of 6.4, indicating accessibility for most medical trainees and clinicians. Robustness and stability were supported by non significant variance in repeated trials (analysis of variance F value 0.61, p greater than 0.1; F value 0.78, p greater than 0.1). A user study with medical students showed that AIPatient provides high fidelity, usability, and educational value, comparable to or better than human simulated patients for history taking. Conclusions: LLM based simulated patient systems can deliver accurate, readable, and reliable medical encounters and show strong potential to transform medical education. |
| title | Simulated patient systems powered by large language model-based AI agents offer potential for transforming medical education |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2409.18924 |