FactEHR: A Dataset for Evaluating Factuality in Clinical Notes Using LLMs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908723836878848 |
|---|---|
| author | Munnangi, Monica Swaminathan, Akshay Fries, Jason Alan Jindal, Jenelle Narayanan, Sanjana Lopez, Ivan Tu, Lucia Chung, Philip Omiye, Jesutofunmi A. Kashyap, Mehr Shah, Nigam |
| author_facet | Munnangi, Monica Swaminathan, Akshay Fries, Jason Alan Jindal, Jenelle Narayanan, Sanjana Lopez, Ivan Tu, Lucia Chung, Philip Omiye, Jesutofunmi A. Kashyap, Mehr Shah, Nigam |
| contents | Verifying and attributing factual claims is essential for the safe and effective use of large language models (LLMs) in healthcare. A core component of factuality evaluation is fact decomposition, the process of breaking down complex clinical statements into fine-grained atomic facts for verification. Recent work has proposed fact decomposition, which uses LLMs to rewrite source text into concise sentences conveying a single piece of information, to facilitate fine-grained fact verification. However, clinical documentation poses unique challenges for fact decomposition due to dense terminology and diverse note types and remains understudied. To address this gap and explore these challenges, we present FactEHR, an NLI dataset consisting of document fact decompositions for 2,168 clinical notes spanning four types from three hospital systems, resulting in 987,266 entailment pairs. We assess the generated facts on different axes, from entailment evaluation of LLMs to a qualitative analysis. Our evaluation, including review by the clinicians, reveals substantial variability in LLM performance for fact decomposition. For example, Gemini-1.5-Flash consistently generates relevant and accurate facts, while Llama-3 8B produces fewer and less consistent outputs. The results underscore the need for better LLM capabilities to support factual verification in clinical text. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_12422 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | FactEHR: A Dataset for Evaluating Factuality in Clinical Notes Using LLMs Munnangi, Monica Swaminathan, Akshay Fries, Jason Alan Jindal, Jenelle Narayanan, Sanjana Lopez, Ivan Tu, Lucia Chung, Philip Omiye, Jesutofunmi A. Kashyap, Mehr Shah, Nigam Computation and Language Verifying and attributing factual claims is essential for the safe and effective use of large language models (LLMs) in healthcare. A core component of factuality evaluation is fact decomposition, the process of breaking down complex clinical statements into fine-grained atomic facts for verification. Recent work has proposed fact decomposition, which uses LLMs to rewrite source text into concise sentences conveying a single piece of information, to facilitate fine-grained fact verification. However, clinical documentation poses unique challenges for fact decomposition due to dense terminology and diverse note types and remains understudied. To address this gap and explore these challenges, we present FactEHR, an NLI dataset consisting of document fact decompositions for 2,168 clinical notes spanning four types from three hospital systems, resulting in 987,266 entailment pairs. We assess the generated facts on different axes, from entailment evaluation of LLMs to a qualitative analysis. Our evaluation, including review by the clinicians, reveals substantial variability in LLM performance for fact decomposition. For example, Gemini-1.5-Flash consistently generates relevant and accurate facts, while Llama-3 8B produces fewer and less consistent outputs. The results underscore the need for better LLM capabilities to support factual verification in clinical text. |
| title | FactEHR: A Dataset for Evaluating Factuality in Clinical Notes Using LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2412.12422 |