Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2605.09272 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910205856448512 |
|---|---|
| author | Shah, Meet Gusdorf, Jason Palepu, Anil Park, Chunjong O'Sullivan, Jack W. Ravi, Vishnu Strother, Tim Dubov, Pavel Rysbek, Aliya Fukuzawa, Toshiyuki Lunts, Yana Freyberg, Jan Chang, Michael B. Raghu, Aniruddh Stutz, David Berlowitz, Devora Papa, Eliseo Cemgil, Taylan Velasquez, JD Chen, Jack Chen, Arthur Fritz, Doug Taylor, Charlie Tregubova, Katya Lim, Jing Rong Green, Richard Mahdavi, Sara Nagda, Mahvish Lee, Jihyeon Schiff, Craig Panait, Liviu Singh, Sukhdeep Liévin, Valentin Barrett, David G. T. Gladman, Hannah Cupani, Anna Pietra, Francesca Okereke, Uchechi Tong, Katherine Meyer, Clemens Rolland, Erwan Sanwalka, Mili Howell, Michael D. Gu, Shixiang Shane Xu, Bibo Ashley, Euan A. Eslami, S. M. Ali Wayne, Gregory Kohli, Pushmeet Natarajan, Vivek Rodman, Adam Karthikesalingam, Alan Tanno, Ryutaro |
| author_facet | Shah, Meet Gusdorf, Jason Palepu, Anil Park, Chunjong O'Sullivan, Jack W. Ravi, Vishnu Strother, Tim Dubov, Pavel Rysbek, Aliya Fukuzawa, Toshiyuki Lunts, Yana Freyberg, Jan Chang, Michael B. Raghu, Aniruddh Stutz, David Berlowitz, Devora Papa, Eliseo Cemgil, Taylan Velasquez, JD Chen, Jack Chen, Arthur Fritz, Doug Taylor, Charlie Tregubova, Katya Lim, Jing Rong Green, Richard Mahdavi, Sara Nagda, Mahvish Lee, Jihyeon Schiff, Craig Panait, Liviu Singh, Sukhdeep Liévin, Valentin Barrett, David G. T. Gladman, Hannah Cupani, Anna Pietra, Francesca Okereke, Uchechi Tong, Katherine Meyer, Clemens Rolland, Erwan Sanwalka, Mili Howell, Michael D. Gu, Shixiang Shane Xu, Bibo Ashley, Euan A. Eslami, S. M. Ali Wayne, Gregory Kohli, Pushmeet Natarajan, Vivek Rodman, Adam Karthikesalingam, Alan Tanno, Ryutaro |
| contents | The practice of medicine relies not only upon skillful dialogue but also on the nuanced exchange and interpretation of rich auditory and visual cues between doctors and patients. Building on the low-latency voice and video processing capabilities of Gemini, we introduce AI co-clinician, a first-of-its-kind conversational AI system utilizing continuous streams of audio-visual data from live patient conversations to inform real-time clinical decisions. Its dual-agent architecture balances deep clinical reasoning with the low latency required for natural dialogue. To assess this system, we implemented a video-based interface emulating telemedicine consultations. We crafted 20 standardized outpatient scenarios requiring proactive real-time auditory and visual reasoning and designed "TelePACES" evaluation criteria alongside case-specific rubrics. In a randomized, interface-blinded, crossover simulation study (n = 120 encounters) with 10 internal medicine residents as patient actors, we compared AI co-clinician with primary care physicians (PCPs), GPT-Realtime, and a baseline agent. AI co-clinician approached PCPs in key TelePACES dimensions, including management plans and differential diagnosis, while significantly outperforming GPT-Realtime across all general criteria. While our agent demonstrated parity with PCPs in case-specific triage measures, physicians maintained superior overall performance in case-specific assessments. Although AI co-clinician marks a significant advance in real-time telemedical AI, gaps remain in physical examination and disease-specific reasoning. Our work shows that text-only approaches fail to capture the true challenges of medical consultation and suggests that high-stakes real-time diagnostic AI is most safely advanced in collaborative, triadic models where AI can be a supportive co-clinician for doctors and patients. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_09272 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Towards Conversational Medical AI with Eyes, Ears and a Voice Shah, Meet Gusdorf, Jason Palepu, Anil Park, Chunjong O'Sullivan, Jack W. Ravi, Vishnu Strother, Tim Dubov, Pavel Rysbek, Aliya Fukuzawa, Toshiyuki Lunts, Yana Freyberg, Jan Chang, Michael B. Raghu, Aniruddh Stutz, David Berlowitz, Devora Papa, Eliseo Cemgil, Taylan Velasquez, JD Chen, Jack Chen, Arthur Fritz, Doug Taylor, Charlie Tregubova, Katya Lim, Jing Rong Green, Richard Mahdavi, Sara Nagda, Mahvish Lee, Jihyeon Schiff, Craig Panait, Liviu Singh, Sukhdeep Liévin, Valentin Barrett, David G. T. Gladman, Hannah Cupani, Anna Pietra, Francesca Okereke, Uchechi Tong, Katherine Meyer, Clemens Rolland, Erwan Sanwalka, Mili Howell, Michael D. Gu, Shixiang Shane Xu, Bibo Ashley, Euan A. Eslami, S. M. Ali Wayne, Gregory Kohli, Pushmeet Natarajan, Vivek Rodman, Adam Karthikesalingam, Alan Tanno, Ryutaro Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition The practice of medicine relies not only upon skillful dialogue but also on the nuanced exchange and interpretation of rich auditory and visual cues between doctors and patients. Building on the low-latency voice and video processing capabilities of Gemini, we introduce AI co-clinician, a first-of-its-kind conversational AI system utilizing continuous streams of audio-visual data from live patient conversations to inform real-time clinical decisions. Its dual-agent architecture balances deep clinical reasoning with the low latency required for natural dialogue. To assess this system, we implemented a video-based interface emulating telemedicine consultations. We crafted 20 standardized outpatient scenarios requiring proactive real-time auditory and visual reasoning and designed "TelePACES" evaluation criteria alongside case-specific rubrics. In a randomized, interface-blinded, crossover simulation study (n = 120 encounters) with 10 internal medicine residents as patient actors, we compared AI co-clinician with primary care physicians (PCPs), GPT-Realtime, and a baseline agent. AI co-clinician approached PCPs in key TelePACES dimensions, including management plans and differential diagnosis, while significantly outperforming GPT-Realtime across all general criteria. While our agent demonstrated parity with PCPs in case-specific triage measures, physicians maintained superior overall performance in case-specific assessments. Although AI co-clinician marks a significant advance in real-time telemedical AI, gaps remain in physical examination and disease-specific reasoning. Our work shows that text-only approaches fail to capture the true challenges of medical consultation and suggests that high-stakes real-time diagnostic AI is most safely advanced in collaborative, triadic models where AI can be a supportive co-clinician for doctors and patients. |
| title | Towards Conversational Medical AI with Eyes, Ears and a Voice |
| topic | Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2605.09272 |