Adoption and Use of LLMs at an Academic Medical Center
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915971399155712 |
|---|---|
| author | Shah, Nigam H. Ambers, Nerissa Pandya, Abby Keyes, Timothy Banda, Juan M. Nallan, Srikar Lugtu, Carlene Trotsyuk, Artem A. Bedi, Suhana Unell, Alyssa Fuentes, Miguel Grolleau, Francois Jain, Sneha S. Chen, Jonathan Dash, Devdutta Char, Danton Sharma, Aditya McElfresh, Duncan Scully, Patrick Kumar, Vishanthan Dennis, Clancy OBrien, Connor Mouniswamy, Satchi Jones, Elvis Jasti, Krishna Lakshmanan, Gunavathi Mannika Akula, Sree Ram Singh, Varun Kumar Rajmanickam, Ramesh Sinha, Sudhir Zhou, Vicky Wang, Xu Mawji, Bilal Ge, Joshua Li, Wencheng Lyons, Travis Helzer, Jarrod Kakkar, Vikas Powar, Ramesh Batara, Darren Cordova, Cheryl Frederick III, William Tang, Olivia Morgan, Phoebe Liang, April S. Ma, Stephen P. Vedak, Shivam Yao, Dong-han Swaminathan, Akshay Kashyap, Mehr Ng, Brian Hellman, Jamie Kotecha, Nikesh Sharp, Christopher Brown, Gretchen Lindmark, Christian Revri, Anurang Pfeffer, Michael A. |
| author_facet | Shah, Nigam H. Ambers, Nerissa Pandya, Abby Keyes, Timothy Banda, Juan M. Nallan, Srikar Lugtu, Carlene Trotsyuk, Artem A. Bedi, Suhana Unell, Alyssa Fuentes, Miguel Grolleau, Francois Jain, Sneha S. Chen, Jonathan Dash, Devdutta Char, Danton Sharma, Aditya McElfresh, Duncan Scully, Patrick Kumar, Vishanthan Dennis, Clancy OBrien, Connor Mouniswamy, Satchi Jones, Elvis Jasti, Krishna Lakshmanan, Gunavathi Mannika Akula, Sree Ram Singh, Varun Kumar Rajmanickam, Ramesh Sinha, Sudhir Zhou, Vicky Wang, Xu Mawji, Bilal Ge, Joshua Li, Wencheng Lyons, Travis Helzer, Jarrod Kakkar, Vikas Powar, Ramesh Batara, Darren Cordova, Cheryl Frederick III, William Tang, Olivia Morgan, Phoebe Liang, April S. Ma, Stephen P. Vedak, Shivam Yao, Dong-han Swaminathan, Akshay Kashyap, Mehr Ng, Brian Hellman, Jamie Kotecha, Nikesh Sharp, Christopher Brown, Gretchen Lindmark, Christian Revri, Anurang Pfeffer, Michael A. |
| contents | While large language models (LLMs) can support clinical documentation needs, standalone tools struggle with "workflow friction" from manual data entry. We developed ChatEHR, a system that enables the use of LLMs with the entire patient timeline spanning several years. ChatEHR enables automations - which are static combinations of prompts and data that perform a fixed task - and interactive use in the electronic health record (EHR) via a user interface (UI). The resulting ability to sift through patient medical records for diverse use-cases such as pre-visit chart review, screening for transfer eligibility, monitoring for surgical site infections, and chart abstraction, redefines LLM use as an institutional capability. This system, accessible after user-training, enables continuous monitoring and evaluation of LLM use.
In 1.5 years, we built 7 automations and 1075 users have trained to become routine users of the UI, engaging in 23,000 sessions in the first 3 months of launch. For automations, being model-agnostic and accessing multiple types of data was essential for matching specific clinical or administrative tasks with the most appropriate LLM. Benchmark-based evaluations proved insufficient for monitoring and evaluation of the UI, requiring new methods to monitor performance. Generation of summaries was the most frequent task in the UI, with an estimated 0.73 hallucinations and 1.60 inaccuracies per generation. The resulting mix of cost savings, time savings, and revenue growth required a value assessment framework to prioritize work as well as quantify the impact of using LLMs. Initial estimates are $6M savings in the first year of use, without quantifying the benefit of the better care offered. Such a "build-from-within" strategy provides an opportunity for health systems to maintain agency via a vendor-agnostic, internally governed LLM platform. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_00074 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Adoption and Use of LLMs at an Academic Medical Center Shah, Nigam H. Ambers, Nerissa Pandya, Abby Keyes, Timothy Banda, Juan M. Nallan, Srikar Lugtu, Carlene Trotsyuk, Artem A. Bedi, Suhana Unell, Alyssa Fuentes, Miguel Grolleau, Francois Jain, Sneha S. Chen, Jonathan Dash, Devdutta Char, Danton Sharma, Aditya McElfresh, Duncan Scully, Patrick Kumar, Vishanthan Dennis, Clancy OBrien, Connor Mouniswamy, Satchi Jones, Elvis Jasti, Krishna Lakshmanan, Gunavathi Mannika Akula, Sree Ram Singh, Varun Kumar Rajmanickam, Ramesh Sinha, Sudhir Zhou, Vicky Wang, Xu Mawji, Bilal Ge, Joshua Li, Wencheng Lyons, Travis Helzer, Jarrod Kakkar, Vikas Powar, Ramesh Batara, Darren Cordova, Cheryl Frederick III, William Tang, Olivia Morgan, Phoebe Liang, April S. Ma, Stephen P. Vedak, Shivam Yao, Dong-han Swaminathan, Akshay Kashyap, Mehr Ng, Brian Hellman, Jamie Kotecha, Nikesh Sharp, Christopher Brown, Gretchen Lindmark, Christian Revri, Anurang Pfeffer, Michael A. Computers and Society Artificial Intelligence While large language models (LLMs) can support clinical documentation needs, standalone tools struggle with "workflow friction" from manual data entry. We developed ChatEHR, a system that enables the use of LLMs with the entire patient timeline spanning several years. ChatEHR enables automations - which are static combinations of prompts and data that perform a fixed task - and interactive use in the electronic health record (EHR) via a user interface (UI). The resulting ability to sift through patient medical records for diverse use-cases such as pre-visit chart review, screening for transfer eligibility, monitoring for surgical site infections, and chart abstraction, redefines LLM use as an institutional capability. This system, accessible after user-training, enables continuous monitoring and evaluation of LLM use. In 1.5 years, we built 7 automations and 1075 users have trained to become routine users of the UI, engaging in 23,000 sessions in the first 3 months of launch. For automations, being model-agnostic and accessing multiple types of data was essential for matching specific clinical or administrative tasks with the most appropriate LLM. Benchmark-based evaluations proved insufficient for monitoring and evaluation of the UI, requiring new methods to monitor performance. Generation of summaries was the most frequent task in the UI, with an estimated 0.73 hallucinations and 1.60 inaccuracies per generation. The resulting mix of cost savings, time savings, and revenue growth required a value assessment framework to prioritize work as well as quantify the impact of using LLMs. Initial estimates are $6M savings in the first year of use, without quantifying the benefit of the better care offered. Such a "build-from-within" strategy provides an opportunity for health systems to maintain agency via a vendor-agnostic, internally governed LLM platform. |
| title | Adoption and Use of LLMs at an Academic Medical Center |
| topic | Computers and Society Artificial Intelligence |
| url | https://arxiv.org/abs/2602.00074 |