Development and Validation of the Provider Documentation Summarization Quality Instrument for Large Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916571328282624 |
|---|---|
| author | Croxford, Emma Gao, Yanjun Pellegrino, Nicholas Wong, Karen K. Wills, Graham First, Elliot Schnier, Miranda Burton, Kyle Ebby, Cris G. Gorskic, Jillian Kalscheur, Matthew Khalil, Samy Pisani, Marie Rubeor, Tyler Stetson, Peter Liao, Frank Goswami, Cherodeep Patterson, Brian Afshar, Majid |
| author_facet | Croxford, Emma Gao, Yanjun Pellegrino, Nicholas Wong, Karen K. Wills, Graham First, Elliot Schnier, Miranda Burton, Kyle Ebby, Cris G. Gorskic, Jillian Kalscheur, Matthew Khalil, Samy Pisani, Marie Rubeor, Tyler Stetson, Peter Liao, Frank Goswami, Cherodeep Patterson, Brian Afshar, Majid |
| contents | As Large Language Models (LLMs) are integrated into electronic health record (EHR) workflows, validated instruments are essential to evaluate their performance before implementation. Existing instruments for provider documentation quality are often unsuitable for the complexities of LLM-generated text and lack validation on real-world data. The Provider Documentation Summarization Quality Instrument (PDSQI-9) was developed to evaluate LLM-generated clinical summaries. Multi-document summaries were generated from real-world EHR data across multiple specialties using several LLMs (GPT-4o, Mixtral 8x7b, and Llama 3-8b). Validation included Pearson correlation for substantive validity, factor analysis and Cronbach's alpha for structural validity, inter-rater reliability (ICC and Krippendorff's alpha) for generalizability, a semi-Delphi process for content validity, and comparisons of high-versus low-quality summaries for discriminant validity. Seven physician raters evaluated 779 summaries and answered 8,329 questions, achieving over 80% power for inter-rater reliability. The PDSQI-9 demonstrated strong internal consistency (Cronbach's alpha = 0.879; 95% CI: 0.867-0.891) and high inter-rater reliability (ICC = 0.867; 95% CI: 0.867-0.868), supporting structural validity and generalizability. Factor analysis identified a 4-factor model explaining 58% of the variance, representing organization, clarity, accuracy, and utility. Substantive validity was supported by correlations between note length and scores for Succinct (rho = -0.200, p = 0.029) and Organized ($ρ= -0.190$, $p = 0.037$). Discriminant validity distinguished high- from low-quality summaries ($p < 0.001$). The PDSQI-9 demonstrates robust construct validity, supporting its use in clinical practice to evaluate LLM-generated summaries and facilitate safer integration of LLMs into healthcare workflows. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_08977 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Development and Validation of the Provider Documentation Summarization Quality Instrument for Large Language Models Croxford, Emma Gao, Yanjun Pellegrino, Nicholas Wong, Karen K. Wills, Graham First, Elliot Schnier, Miranda Burton, Kyle Ebby, Cris G. Gorskic, Jillian Kalscheur, Matthew Khalil, Samy Pisani, Marie Rubeor, Tyler Stetson, Peter Liao, Frank Goswami, Cherodeep Patterson, Brian Afshar, Majid Artificial Intelligence As Large Language Models (LLMs) are integrated into electronic health record (EHR) workflows, validated instruments are essential to evaluate their performance before implementation. Existing instruments for provider documentation quality are often unsuitable for the complexities of LLM-generated text and lack validation on real-world data. The Provider Documentation Summarization Quality Instrument (PDSQI-9) was developed to evaluate LLM-generated clinical summaries. Multi-document summaries were generated from real-world EHR data across multiple specialties using several LLMs (GPT-4o, Mixtral 8x7b, and Llama 3-8b). Validation included Pearson correlation for substantive validity, factor analysis and Cronbach's alpha for structural validity, inter-rater reliability (ICC and Krippendorff's alpha) for generalizability, a semi-Delphi process for content validity, and comparisons of high-versus low-quality summaries for discriminant validity. Seven physician raters evaluated 779 summaries and answered 8,329 questions, achieving over 80% power for inter-rater reliability. The PDSQI-9 demonstrated strong internal consistency (Cronbach's alpha = 0.879; 95% CI: 0.867-0.891) and high inter-rater reliability (ICC = 0.867; 95% CI: 0.867-0.868), supporting structural validity and generalizability. Factor analysis identified a 4-factor model explaining 58% of the variance, representing organization, clarity, accuracy, and utility. Substantive validity was supported by correlations between note length and scores for Succinct (rho = -0.200, p = 0.029) and Organized ($ρ= -0.190$, $p = 0.037$). Discriminant validity distinguished high- from low-quality summaries ($p < 0.001$). The PDSQI-9 demonstrates robust construct validity, supporting its use in clinical practice to evaluate LLM-generated summaries and facilitate safer integration of LLMs into healthcare workflows. |
| title | Development and Validation of the Provider Documentation Summarization Quality Instrument for Large Language Models |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2501.08977 |