FactPICO: Factuality Evaluation for Plain Language Summarization of Medical Evidence
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917684742979584 |
|---|---|
| author | Joseph, Sebastian Antony Chen, Lily Trienes, Jan Göke, Hannah Louisa Coers, Monika Xu, Wei Wallace, Byron C Li, Junyi Jessy |
| author_facet | Joseph, Sebastian Antony Chen, Lily Trienes, Jan Göke, Hannah Louisa Coers, Monika Xu, Wei Wallace, Byron C Li, Junyi Jessy |
| contents | Plain language summarization with LLMs can be useful for improving textual accessibility of technical content. But how factual are these summaries in a high-stakes domain like medicine? This paper presents FactPICO, a factuality benchmark for plain language summarization of medical texts describing randomized controlled trials (RCTs), which are the basis of evidence-based medicine and can directly inform patient treatment. FactPICO consists of 345 plain language summaries of RCT abstracts generated from three LLMs (i.e., GPT-4, Llama-2, and Alpaca), with fine-grained evaluation and natural language rationales from experts. We assess the factuality of critical elements of RCTs in those summaries: Populations, Interventions, Comparators, Outcomes (PICO), as well as the reported findings concerning these. We also evaluate the correctness of the extra information (e.g., explanations) added by LLMs. Using FactPICO, we benchmark a range of existing factuality metrics, including the newly devised ones based on LLMs. We find that plain language summarization of medical evidence is still challenging, especially when balancing between simplicity and factuality, and that existing metrics correlate poorly with expert judgments on the instance level. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_11456 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | FactPICO: Factuality Evaluation for Plain Language Summarization of Medical Evidence Joseph, Sebastian Antony Chen, Lily Trienes, Jan Göke, Hannah Louisa Coers, Monika Xu, Wei Wallace, Byron C Li, Junyi Jessy Computation and Language Plain language summarization with LLMs can be useful for improving textual accessibility of technical content. But how factual are these summaries in a high-stakes domain like medicine? This paper presents FactPICO, a factuality benchmark for plain language summarization of medical texts describing randomized controlled trials (RCTs), which are the basis of evidence-based medicine and can directly inform patient treatment. FactPICO consists of 345 plain language summaries of RCT abstracts generated from three LLMs (i.e., GPT-4, Llama-2, and Alpaca), with fine-grained evaluation and natural language rationales from experts. We assess the factuality of critical elements of RCTs in those summaries: Populations, Interventions, Comparators, Outcomes (PICO), as well as the reported findings concerning these. We also evaluate the correctness of the extra information (e.g., explanations) added by LLMs. Using FactPICO, we benchmark a range of existing factuality metrics, including the newly devised ones based on LLMs. We find that plain language summarization of medical evidence is still challenging, especially when balancing between simplicity and factuality, and that existing metrics correlate poorly with expert judgments on the instance level. |
| title | FactPICO: Factuality Evaluation for Plain Language Summarization of Medical Evidence |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2402.11456 |