Advancing Conversational Diagnostic AI with Multimodal Reasoning
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909604374380544 |
|---|---|
| author | Saab, Khaled Freyberg, Jan Park, Chunjong Strother, Tim Cheng, Yong Weng, Wei-Hung Barrett, David G. T. Stutz, David Tomasev, Nenad Palepu, Anil Liévin, Valentin Sharma, Yash Ruparel, Roma Ahmed, Abdullah Vedadi, Elahe Kanada, Kimberly Hughes, Cian Liu, Yun Brown, Geoff Gao, Yang Li, Sean Mahdavi, S. Sara Manyika, James Chou, Katherine Matias, Yossi Hassidim, Avinatan Webster, Dale R. Kohli, Pushmeet Eslami, S. M. Ali Barral, Joëlle Rodman, Adam Natarajan, Vivek Schaekermann, Mike Tu, Tao Karthikesalingam, Alan Tanno, Ryutaro |
| author_facet | Saab, Khaled Freyberg, Jan Park, Chunjong Strother, Tim Cheng, Yong Weng, Wei-Hung Barrett, David G. T. Stutz, David Tomasev, Nenad Palepu, Anil Liévin, Valentin Sharma, Yash Ruparel, Roma Ahmed, Abdullah Vedadi, Elahe Kanada, Kimberly Hughes, Cian Liu, Yun Brown, Geoff Gao, Yang Li, Sean Mahdavi, S. Sara Manyika, James Chou, Katherine Matias, Yossi Hassidim, Avinatan Webster, Dale R. Kohli, Pushmeet Eslami, S. M. Ali Barral, Joëlle Rodman, Adam Natarajan, Vivek Schaekermann, Mike Tu, Tao Karthikesalingam, Alan Tanno, Ryutaro |
| contents | Large Language Models (LLMs) have demonstrated great potential for conducting diagnostic conversations but evaluation has been largely limited to language-only interactions, deviating from the real-world requirements of remote care delivery. Instant messaging platforms permit clinicians and patients to upload and discuss multimodal medical artifacts seamlessly in medical consultation, but the ability of LLMs to reason over such data while preserving other attributes of competent diagnostic conversation remains unknown. Here we advance the conversational diagnosis and management performance of the Articulate Medical Intelligence Explorer (AMIE) through a new capability to gather and interpret multimodal data, and reason about this precisely during consultations. Leveraging Gemini 2.0 Flash, our system implements a state-aware dialogue framework, where conversation flow is dynamically controlled by intermediate model outputs reflecting patient states and evolving diagnoses. Follow-up questions are strategically directed by uncertainty in such patient states, leading to a more structured multimodal history-taking process that emulates experienced clinicians. We compared AMIE to primary care physicians (PCPs) in a randomized, blinded, OSCE-style study of chat-based consultations with patient actors. We constructed 105 evaluation scenarios using artifacts like smartphone skin photos, ECGs, and PDFs of clinical documents across diverse conditions and demographics. Our rubric assessed multimodal capabilities and other clinically meaningful axes like history-taking, diagnostic accuracy, management reasoning, communication, and empathy. Specialist evaluation showed AMIE to be superior to PCPs on 7/9 multimodal and 29/32 non-multimodal axes (including diagnostic accuracy). The results show clear progress in multimodal conversational diagnostic AI, but real-world translation needs further research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_04653 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Advancing Conversational Diagnostic AI with Multimodal Reasoning Saab, Khaled Freyberg, Jan Park, Chunjong Strother, Tim Cheng, Yong Weng, Wei-Hung Barrett, David G. T. Stutz, David Tomasev, Nenad Palepu, Anil Liévin, Valentin Sharma, Yash Ruparel, Roma Ahmed, Abdullah Vedadi, Elahe Kanada, Kimberly Hughes, Cian Liu, Yun Brown, Geoff Gao, Yang Li, Sean Mahdavi, S. Sara Manyika, James Chou, Katherine Matias, Yossi Hassidim, Avinatan Webster, Dale R. Kohli, Pushmeet Eslami, S. M. Ali Barral, Joëlle Rodman, Adam Natarajan, Vivek Schaekermann, Mike Tu, Tao Karthikesalingam, Alan Tanno, Ryutaro Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning Large Language Models (LLMs) have demonstrated great potential for conducting diagnostic conversations but evaluation has been largely limited to language-only interactions, deviating from the real-world requirements of remote care delivery. Instant messaging platforms permit clinicians and patients to upload and discuss multimodal medical artifacts seamlessly in medical consultation, but the ability of LLMs to reason over such data while preserving other attributes of competent diagnostic conversation remains unknown. Here we advance the conversational diagnosis and management performance of the Articulate Medical Intelligence Explorer (AMIE) through a new capability to gather and interpret multimodal data, and reason about this precisely during consultations. Leveraging Gemini 2.0 Flash, our system implements a state-aware dialogue framework, where conversation flow is dynamically controlled by intermediate model outputs reflecting patient states and evolving diagnoses. Follow-up questions are strategically directed by uncertainty in such patient states, leading to a more structured multimodal history-taking process that emulates experienced clinicians. We compared AMIE to primary care physicians (PCPs) in a randomized, blinded, OSCE-style study of chat-based consultations with patient actors. We constructed 105 evaluation scenarios using artifacts like smartphone skin photos, ECGs, and PDFs of clinical documents across diverse conditions and demographics. Our rubric assessed multimodal capabilities and other clinically meaningful axes like history-taking, diagnostic accuracy, management reasoning, communication, and empathy. Specialist evaluation showed AMIE to be superior to PCPs on 7/9 multimodal and 29/32 non-multimodal axes (including diagnostic accuracy). The results show clear progress in multimodal conversational diagnostic AI, but real-world translation needs further research. |
| title | Advancing Conversational Diagnostic AI with Multimodal Reasoning |
| topic | Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2505.04653 |