Can artificial intelligence pass the test? Evaluating chatbot scores on pediatric gastroenterology board‐style questions
Fuente:
Wiley Open Access
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Artículo Open Access |
| Published: |
Wiley
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1867002903024107520 |
|---|---|
| author | Annette G. Roberts Reshma Patel Sharmilaa Babu Matthew M. Engelhard Rachel G. Greenberg Arun Ajmera |
| author_facet | Annette G. Roberts Reshma Patel Sharmilaa Babu Matthew M. Engelhard Rachel G. Greenberg Arun Ajmera Annette G. Roberts Reshma Patel Sharmilaa Babu Matthew M. Engelhard Rachel G. Greenberg Arun Ajmera |
| collection | Wiley Open Access |
| contents | Can artificial intelligence pass the test? Evaluating chatbot scores on pediatric gastroenterology board‐style questions Annette G. Roberts Reshma Patel Sharmilaa Babu Matthew M. Engelhard Rachel G. Greenberg Arun Ajmera JPGN Reports Abstract Objectives The American Academy of Pediatrics (AAP) Pediatrics Review and Education Program (PREP)® Gastroenterology (GI) Self‐Assessments help pediatric gastroenterologists and trainees prepare for subspecialty board exams by providing peer‐reviewed questions and critiques based on American Board of Pediatrics content specifications. These assessments test knowledge of material aligned with the pediatric gastroenterology board exams. While artificial intelligence (AI) chatbots have passed various medical board exams, their ability to pass the pediatric GI boards remains untested. This study assesses the performance of Microsoft Copilot and OpenAI ChatGPT‐3.5 and 4o on the 2022‐2024 AAP PREP® GI Self‐Assessments. Methods A total of 216 AAP PREP® GI Self‐Assessment questions from 2022 to 2024 were entered into three AI chatbots (Microsoft Copilot, OpenAI ChatGPT‐3.5, and ChatGPT‐4o). Scores were compared with the passing score (> 65%) and first‐time test takers' scores from the AAP for 2022–2024. Results OpenAI ChatGPT‐4o and Microsoft Copilot scored above 65% (pass) on all three PREP® GI Self‐Assessments from 2022 to 2024. OpenAI ChatGPT‐3.5 passed the 2023 and 2024 assessments but did not pass the 2022 assessment. The chatbots collectively scored best in anatomy, motility, and mouth and esophageal disorders, and scored poorly in physiology, pharmacology, liver, stomach and duodenum disorders. Conclusions OpenAI ChatGPT‐4o and Microsoft Copilot consistently passed the PREP® GI Self‐Assessments from 2022 to 2024, showing potential for good performance on the pediatric GI boards. OpenAI ChatGPT‐3.5 had limitations, passing only the 2023 and 2024 assessments. Overall, advanced AI chatbots show potential to pass the Pediatric GI board exam. 10.1002/jpr3.70121 http://creativecommons.org/licenses/by-nc/4.0/ |
| doi_str_mv | 10.1002/jpr3.70121 |
| format | Artículo Open Access |
| id | wiley_oa_10_1002_jpr3_70121 |
| institution | Wiley Open Access |
| license_str_mv | http://creativecommons.org/licenses/by-nc/4.0/ |
| publishDate | 2025 |
| publisher | Wiley |
| record_format | wiley_oa |
| spellingShingle | Can artificial intelligence pass the test? Evaluating chatbot scores on pediatric gastroenterology board‐style questions Annette G. Roberts Reshma Patel Sharmilaa Babu Matthew M. Engelhard Rachel G. Greenberg Arun Ajmera JPGN Reports Can artificial intelligence pass the test? Evaluating chatbot scores on pediatric gastroenterology board‐style questions Annette G. Roberts Reshma Patel Sharmilaa Babu Matthew M. Engelhard Rachel G. Greenberg Arun Ajmera JPGN Reports Abstract Objectives The American Academy of Pediatrics (AAP) Pediatrics Review and Education Program (PREP)® Gastroenterology (GI) Self‐Assessments help pediatric gastroenterologists and trainees prepare for subspecialty board exams by providing peer‐reviewed questions and critiques based on American Board of Pediatrics content specifications. These assessments test knowledge of material aligned with the pediatric gastroenterology board exams. While artificial intelligence (AI) chatbots have passed various medical board exams, their ability to pass the pediatric GI boards remains untested. This study assesses the performance of Microsoft Copilot and OpenAI ChatGPT‐3.5 and 4o on the 2022‐2024 AAP PREP® GI Self‐Assessments. Methods A total of 216 AAP PREP® GI Self‐Assessment questions from 2022 to 2024 were entered into three AI chatbots (Microsoft Copilot, OpenAI ChatGPT‐3.5, and ChatGPT‐4o). Scores were compared with the passing score (> 65%) and first‐time test takers' scores from the AAP for 2022–2024. Results OpenAI ChatGPT‐4o and Microsoft Copilot scored above 65% (pass) on all three PREP® GI Self‐Assessments from 2022 to 2024. OpenAI ChatGPT‐3.5 passed the 2023 and 2024 assessments but did not pass the 2022 assessment. The chatbots collectively scored best in anatomy, motility, and mouth and esophageal disorders, and scored poorly in physiology, pharmacology, liver, stomach and duodenum disorders. Conclusions OpenAI ChatGPT‐4o and Microsoft Copilot consistently passed the PREP® GI Self‐Assessments from 2022 to 2024, showing potential for good performance on the pediatric GI boards. OpenAI ChatGPT‐3.5 had limitations, passing only the 2023 and 2024 assessments. Overall, advanced AI chatbots show potential to pass the Pediatric GI board exam. 10.1002/jpr3.70121 http://creativecommons.org/licenses/by-nc/4.0/ |
| title | Can artificial intelligence pass the test? Evaluating chatbot scores on pediatric gastroenterology board‐style questions |
| topic | JPGN Reports |
| url | https://onlinelibrary.wiley.com/doi/10.1002/jpr3.70121 |