Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicine
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913487559589888 |
|---|---|
| author | Jin, Qiao Chen, Fangyuan Zhou, Yiliang Xu, Ziyang Cheung, Justin M. Chen, Robert Summers, Ronald M. Rousseau, Justin F. Ni, Peiyun Landsman, Marc J Baxter, Sally L. Al'Aref, Subhi J. Li, Yijia Chen, Alex Brejt, Josef A. Chiang, Michael F. Peng, Yifan Lu, Zhiyong |
| author_facet | Jin, Qiao Chen, Fangyuan Zhou, Yiliang Xu, Ziyang Cheung, Justin M. Chen, Robert Summers, Ronald M. Rousseau, Justin F. Ni, Peiyun Landsman, Marc J Baxter, Sally L. Al'Aref, Subhi J. Li, Yijia Chen, Alex Brejt, Josef A. Chiang, Michael F. Peng, Yifan Lu, Zhiyong |
| contents | Recent studies indicate that Generative Pre-trained Transformer 4 with Vision (GPT-4V) outperforms human physicians in medical challenge tasks. However, these evaluations primarily focused on the accuracy of multi-choice questions alone. Our study extends the current scope by conducting a comprehensive analysis of GPT-4V's rationales of image comprehension, recall of medical knowledge, and step-by-step multimodal reasoning when solving New England Journal of Medicine (NEJM) Image Challenges - an imaging quiz designed to test the knowledge and diagnostic capabilities of medical professionals. Evaluation results confirmed that GPT-4V performs comparatively to human physicians regarding multi-choice accuracy (81.6% vs. 77.8%). GPT-4V also performs well in cases where physicians incorrectly answer, with over 78% accuracy. However, we discovered that GPT-4V frequently presents flawed rationales in cases where it makes the correct final choices (35.5%), most prominent in image comprehension (27.2%). Regardless of GPT-4V's high accuracy in multi-choice questions, our findings emphasize the necessity for further in-depth evaluations of its rationales before integrating such multimodal AI models into clinical workflows. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_08396 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicine Jin, Qiao Chen, Fangyuan Zhou, Yiliang Xu, Ziyang Cheung, Justin M. Chen, Robert Summers, Ronald M. Rousseau, Justin F. Ni, Peiyun Landsman, Marc J Baxter, Sally L. Al'Aref, Subhi J. Li, Yijia Chen, Alex Brejt, Josef A. Chiang, Michael F. Peng, Yifan Lu, Zhiyong Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Recent studies indicate that Generative Pre-trained Transformer 4 with Vision (GPT-4V) outperforms human physicians in medical challenge tasks. However, these evaluations primarily focused on the accuracy of multi-choice questions alone. Our study extends the current scope by conducting a comprehensive analysis of GPT-4V's rationales of image comprehension, recall of medical knowledge, and step-by-step multimodal reasoning when solving New England Journal of Medicine (NEJM) Image Challenges - an imaging quiz designed to test the knowledge and diagnostic capabilities of medical professionals. Evaluation results confirmed that GPT-4V performs comparatively to human physicians regarding multi-choice accuracy (81.6% vs. 77.8%). GPT-4V also performs well in cases where physicians incorrectly answer, with over 78% accuracy. However, we discovered that GPT-4V frequently presents flawed rationales in cases where it makes the correct final choices (35.5%), most prominent in image comprehension (27.2%). Regardless of GPT-4V's high accuracy in multi-choice questions, our findings emphasize the necessity for further in-depth evaluations of its rationales before integrating such multimodal AI models into clinical workflows. |
| title | Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicine |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2401.08396 |