GPT-4's assessment of its performance in a USMLE-based case study
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914730069721088 |
|---|---|
| author | Dhakal, Uttam Singh, Aniket Kumar Devkota, Suman Sapkota, Yogesh Lamichhane, Bishal Paudyal, Suprinsa Dhakal, Chandra |
| author_facet | Dhakal, Uttam Singh, Aniket Kumar Devkota, Suman Sapkota, Yogesh Lamichhane, Bishal Paudyal, Suprinsa Dhakal, Chandra |
| contents | This study investigates GPT-4's assessment of its performance in healthcare applications. A simple prompting technique was used to prompt the LLM with questions taken from the United States Medical Licensing Examination (USMLE) questionnaire and it was tasked to evaluate its confidence score before posing the question and after asking the question. The questionnaire was categorized into two groups-questions with feedback (WF) and questions with no feedback(NF) post-question. The model was asked to provide absolute and relative confidence scores before and after each question. The experimental findings were analyzed using statistical tools to study the variability of confidence in WF and NF groups. Additionally, a sequential analysis was conducted to observe the performance variation for the WF and NF groups. Results indicate that feedback influences relative confidence but doesn't consistently increase or decrease it. Understanding the performance of LLM is paramount in exploring its utility in sensitive areas like healthcare. This study contributes to the ongoing discourse on the reliability of AI, particularly of LLMs like GPT-4, within healthcare, offering insights into how feedback mechanisms might be optimized to enhance AI-assisted medical education and decision support. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_09654 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | GPT-4's assessment of its performance in a USMLE-based case study Dhakal, Uttam Singh, Aniket Kumar Devkota, Suman Sapkota, Yogesh Lamichhane, Bishal Paudyal, Suprinsa Dhakal, Chandra Artificial Intelligence Computation and Language Human-Computer Interaction Multiagent Systems Machine Learning This study investigates GPT-4's assessment of its performance in healthcare applications. A simple prompting technique was used to prompt the LLM with questions taken from the United States Medical Licensing Examination (USMLE) questionnaire and it was tasked to evaluate its confidence score before posing the question and after asking the question. The questionnaire was categorized into two groups-questions with feedback (WF) and questions with no feedback(NF) post-question. The model was asked to provide absolute and relative confidence scores before and after each question. The experimental findings were analyzed using statistical tools to study the variability of confidence in WF and NF groups. Additionally, a sequential analysis was conducted to observe the performance variation for the WF and NF groups. Results indicate that feedback influences relative confidence but doesn't consistently increase or decrease it. Understanding the performance of LLM is paramount in exploring its utility in sensitive areas like healthcare. This study contributes to the ongoing discourse on the reliability of AI, particularly of LLMs like GPT-4, within healthcare, offering insights into how feedback mechanisms might be optimized to enhance AI-assisted medical education and decision support. |
| title | GPT-4's assessment of its performance in a USMLE-based case study |
| topic | Artificial Intelligence Computation and Language Human-Computer Interaction Multiagent Systems Machine Learning |
| url | https://arxiv.org/abs/2402.09654 |