Asking the Right Questions: Benchmarking Large Language Models in the Development of Clinical Consultation Templates
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915612528214016 |
|---|---|
| author | McCoy, Liam G. Haredasht, Fateme Nateghi Chopra, Kanav Wu, David Wu, David JH Conteh, Abass Khemani, Sarita Maharaj, Saloni Kumar Ravi, Vishnu Pahwa, Arth Weng, Yingjie Rosengaus, Leah Giang, Lena Li, Kelvin Zhenghao Jee, Olivia Shirvani, Daniel Goh, Ethan Chen, Jonathan H. |
| author_facet | McCoy, Liam G. Haredasht, Fateme Nateghi Chopra, Kanav Wu, David Wu, David JH Conteh, Abass Khemani, Sarita Maharaj, Saloni Kumar Ravi, Vishnu Pahwa, Arth Weng, Yingjie Rosengaus, Leah Giang, Lena Li, Kelvin Zhenghao Jee, Olivia Shirvani, Daniel Goh, Ethan Chen, Jonathan H. |
| contents | This study evaluates the capacity of large language models (LLMs) to generate structured clinical consultation templates for electronic consultation. Using 145 expert-crafted templates developed and routinely used by Stanford's eConsult team, we assess frontier models -- including o3, GPT-4o, Kimi K2, Claude 4 Sonnet, Llama 3 70B, and Gemini 2.5 Pro -- for their ability to produce clinically coherent, concise, and prioritized clinical question schemas. Through a multi-agent pipeline combining prompt optimization, semantic autograding, and prioritization analysis, we show that while models like o3 achieve high comprehensiveness (up to 92.2\%), they consistently generate excessively long templates and fail to correctly prioritize the most clinically important questions under length constraints. Performance varies across specialties, with significant degradation in narrative-driven fields such as psychiatry and pain medicine. Our findings demonstrate that LLMs can enhance structured clinical information exchange between physicians, while highlighting the need for more robust evaluation methods that capture a model's ability to prioritize clinically salient information within the time constraints of real-world physician communication. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_01159 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Asking the Right Questions: Benchmarking Large Language Models in the Development of Clinical Consultation Templates McCoy, Liam G. Haredasht, Fateme Nateghi Chopra, Kanav Wu, David Wu, David JH Conteh, Abass Khemani, Sarita Maharaj, Saloni Kumar Ravi, Vishnu Pahwa, Arth Weng, Yingjie Rosengaus, Leah Giang, Lena Li, Kelvin Zhenghao Jee, Olivia Shirvani, Daniel Goh, Ethan Chen, Jonathan H. Computation and Language Artificial Intelligence This study evaluates the capacity of large language models (LLMs) to generate structured clinical consultation templates for electronic consultation. Using 145 expert-crafted templates developed and routinely used by Stanford's eConsult team, we assess frontier models -- including o3, GPT-4o, Kimi K2, Claude 4 Sonnet, Llama 3 70B, and Gemini 2.5 Pro -- for their ability to produce clinically coherent, concise, and prioritized clinical question schemas. Through a multi-agent pipeline combining prompt optimization, semantic autograding, and prioritization analysis, we show that while models like o3 achieve high comprehensiveness (up to 92.2\%), they consistently generate excessively long templates and fail to correctly prioritize the most clinically important questions under length constraints. Performance varies across specialties, with significant degradation in narrative-driven fields such as psychiatry and pain medicine. Our findings demonstrate that LLMs can enhance structured clinical information exchange between physicians, while highlighting the need for more robust evaluation methods that capture a model's ability to prioritize clinically salient information within the time constraints of real-world physician communication. |
| title | Asking the Right Questions: Benchmarking Large Language Models in the Development of Clinical Consultation Templates |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2508.01159 |