VeriFact: Verifying Facts in LLM-Generated Clinical Text with Electronic Health Records
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917904158556160 |
|---|---|
| author | Chung, Philip Swaminathan, Akshay Goodell, Alex J. Kim, Yeasul Reincke, S. Momsen Han, Lichy Deverett, Ben Sadeghi, Mohammad Amin Ariss, Abdel-Badih Ghanem, Marc Seong, David Lee, Andrew A. Coombes, Caitlin E. Bradshaw, Brad Sufian, Mahir A. Hong, Hyo Jung Nguyen, Teresa P. Rasouli, Mohammad R. Kamra, Komal Burbridge, Mark A. McAvoy, James C. Saffary, Roya Ma, Stephen P. Dash, Dev Xie, James Wang, Ellen Y. Schmiesing, Clifford A. Shah, Nigam Aghaeepour, Nima |
| author_facet | Chung, Philip Swaminathan, Akshay Goodell, Alex J. Kim, Yeasul Reincke, S. Momsen Han, Lichy Deverett, Ben Sadeghi, Mohammad Amin Ariss, Abdel-Badih Ghanem, Marc Seong, David Lee, Andrew A. Coombes, Caitlin E. Bradshaw, Brad Sufian, Mahir A. Hong, Hyo Jung Nguyen, Teresa P. Rasouli, Mohammad R. Kamra, Komal Burbridge, Mark A. McAvoy, James C. Saffary, Roya Ma, Stephen P. Dash, Dev Xie, James Wang, Ellen Y. Schmiesing, Clifford A. Shah, Nigam Aghaeepour, Nima |
| contents | Methods to ensure factual accuracy of text generated by large language models (LLM) in clinical medicine are lacking. VeriFact is an artificial intelligence system that combines retrieval-augmented generation and LLM-as-a-Judge to verify whether LLM-generated text is factually supported by a patient's medical history based on their electronic health record (EHR). To evaluate this system, we introduce VeriFact-BHC, a new dataset that decomposes Brief Hospital Course narratives from discharge summaries into a set of simple statements with clinician annotations for whether each statement is supported by the patient's EHR clinical notes. Whereas highest agreement between clinicians was 88.5%, VeriFact achieves up to 92.7% agreement when compared to a denoised and adjudicated average human clinican ground truth, suggesting that VeriFact exceeds the average clinician's ability to fact-check text against a patient's medical record. VeriFact may accelerate the development of LLM-based EHR applications by removing current evaluation bottlenecks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_16672 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | VeriFact: Verifying Facts in LLM-Generated Clinical Text with Electronic Health Records Chung, Philip Swaminathan, Akshay Goodell, Alex J. Kim, Yeasul Reincke, S. Momsen Han, Lichy Deverett, Ben Sadeghi, Mohammad Amin Ariss, Abdel-Badih Ghanem, Marc Seong, David Lee, Andrew A. Coombes, Caitlin E. Bradshaw, Brad Sufian, Mahir A. Hong, Hyo Jung Nguyen, Teresa P. Rasouli, Mohammad R. Kamra, Komal Burbridge, Mark A. McAvoy, James C. Saffary, Roya Ma, Stephen P. Dash, Dev Xie, James Wang, Ellen Y. Schmiesing, Clifford A. Shah, Nigam Aghaeepour, Nima Artificial Intelligence Computation and Language Information Retrieval Logic in Computer Science Methods to ensure factual accuracy of text generated by large language models (LLM) in clinical medicine are lacking. VeriFact is an artificial intelligence system that combines retrieval-augmented generation and LLM-as-a-Judge to verify whether LLM-generated text is factually supported by a patient's medical history based on their electronic health record (EHR). To evaluate this system, we introduce VeriFact-BHC, a new dataset that decomposes Brief Hospital Course narratives from discharge summaries into a set of simple statements with clinician annotations for whether each statement is supported by the patient's EHR clinical notes. Whereas highest agreement between clinicians was 88.5%, VeriFact achieves up to 92.7% agreement when compared to a denoised and adjudicated average human clinican ground truth, suggesting that VeriFact exceeds the average clinician's ability to fact-check text against a patient's medical record. VeriFact may accelerate the development of LLM-based EHR applications by removing current evaluation bottlenecks. |
| title | VeriFact: Verifying Facts in LLM-Generated Clinical Text with Electronic Health Records |
| topic | Artificial Intelligence Computation and Language Information Retrieval Logic in Computer Science |
| url | https://arxiv.org/abs/2501.16672 |