Evaluating Generative AI as an Educational Tool for Radiology Resident Report Drafting
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911248943153152 |
|---|---|
| author | Verdone, Antonio Cardall, Aidan Siddiqui, Fardeen Nashawaty, Motaz Rigau, Danielle Kwon, Youngjoon Yousef, Mira Patel, Shalin Kieturakis, Alex Kim, Eric Heacock, Laura Reig, Beatriu Shen, Yiqiu |
| author_facet | Verdone, Antonio Cardall, Aidan Siddiqui, Fardeen Nashawaty, Motaz Rigau, Danielle Kwon, Youngjoon Yousef, Mira Patel, Shalin Kieturakis, Alex Kim, Eric Heacock, Laura Reig, Beatriu Shen, Yiqiu |
| contents | Objective: Radiology residents require timely, personalized feedback to develop accurate image analysis and reporting skills. Increasing clinical workload often limits attendings' ability to provide guidance. This study evaluates a HIPAA-compliant GPT-4o system that delivers automated feedback on breast imaging reports drafted by residents in real clinical settings.
Methods: We analyzed 5,000 resident-attending report pairs from routine practice at a multi-site U.S. health system. GPT-4o was prompted with clinical instructions to identify common errors and provide feedback. A reader study using 100 report pairs was conducted. Four attending radiologists and four residents independently reviewed each pair, determined whether predefined error types were present, and rated GPT-4o's feedback as helpful or not. Agreement between GPT and readers was assessed using percent match. Inter-reader reliability was measured with Krippendorff's alpha. Educational value was measured as the proportion of cases rated helpful.
Results: Three common error types were identified: (1) omission or addition of key findings, (2) incorrect use or omission of technical descriptors, and (3) final assessment inconsistent with findings. GPT-4o showed strong agreement with attending consensus: 90.5%, 78.3%, and 90.4% across error types. Inter-reader reliability showed moderate variability (α = 0.767, 0.595, 0.567), and replacing a human reader with GPT-4o did not significantly affect agreement (Δ = -0.004 to 0.002). GPT's feedback was rated helpful in most cases: 89.8%, 83.0%, and 92.0%.
Discussion: ChatGPT-4o can reliably identify key educational errors. It may serve as a scalable tool to support radiology education. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_02839 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Evaluating Generative AI as an Educational Tool for Radiology Resident Report Drafting Verdone, Antonio Cardall, Aidan Siddiqui, Fardeen Nashawaty, Motaz Rigau, Danielle Kwon, Youngjoon Yousef, Mira Patel, Shalin Kieturakis, Alex Kim, Eric Heacock, Laura Reig, Beatriu Shen, Yiqiu Human-Computer Interaction Artificial Intelligence Computers and Society Objective: Radiology residents require timely, personalized feedback to develop accurate image analysis and reporting skills. Increasing clinical workload often limits attendings' ability to provide guidance. This study evaluates a HIPAA-compliant GPT-4o system that delivers automated feedback on breast imaging reports drafted by residents in real clinical settings. Methods: We analyzed 5,000 resident-attending report pairs from routine practice at a multi-site U.S. health system. GPT-4o was prompted with clinical instructions to identify common errors and provide feedback. A reader study using 100 report pairs was conducted. Four attending radiologists and four residents independently reviewed each pair, determined whether predefined error types were present, and rated GPT-4o's feedback as helpful or not. Agreement between GPT and readers was assessed using percent match. Inter-reader reliability was measured with Krippendorff's alpha. Educational value was measured as the proportion of cases rated helpful. Results: Three common error types were identified: (1) omission or addition of key findings, (2) incorrect use or omission of technical descriptors, and (3) final assessment inconsistent with findings. GPT-4o showed strong agreement with attending consensus: 90.5%, 78.3%, and 90.4% across error types. Inter-reader reliability showed moderate variability (α = 0.767, 0.595, 0.567), and replacing a human reader with GPT-4o did not significantly affect agreement (Δ = -0.004 to 0.002). GPT's feedback was rated helpful in most cases: 89.8%, 83.0%, and 92.0%. Discussion: ChatGPT-4o can reliably identify key educational errors. It may serve as a scalable tool to support radiology education. |
| title | Evaluating Generative AI as an Educational Tool for Radiology Resident Report Drafting |
| topic | Human-Computer Interaction Artificial Intelligence Computers and Society |
| url | https://arxiv.org/abs/2511.02839 |