STORYSUMM: Evaluating Faithfulness in Story Summarization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916668815441920 |
|---|---|
| author | Subbiah, Melanie Ladhak, Faisal Mishra, Akankshya Adams, Griffin Chilton, Lydia B. McKeown, Kathleen |
| author_facet | Subbiah, Melanie Ladhak, Faisal Mishra, Akankshya Adams, Griffin Chilton, Lydia B. McKeown, Kathleen |
| contents | Human evaluation has been the gold standard for checking faithfulness in abstractive summarization. However, with a challenging source domain like narrative, multiple annotators can agree a summary is faithful, while missing details that are obvious errors only once pointed out. We therefore introduce a new dataset, STORYSUMM, comprising LLM summaries of short stories with localized faithfulness labels and error explanations. This benchmark is for evaluation methods, testing whether a given method can detect challenging inconsistencies. Using this dataset, we first show that any one human annotation protocol is likely to miss inconsistencies, and we advocate for pursuing a range of methods when establishing ground truth for a summarization dataset. We finally test recent automatic metrics and find that none of them achieve more than 70% balanced accuracy on this task, demonstrating that it is a challenging benchmark for future work in faithfulness evaluation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_06501 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | STORYSUMM: Evaluating Faithfulness in Story Summarization Subbiah, Melanie Ladhak, Faisal Mishra, Akankshya Adams, Griffin Chilton, Lydia B. McKeown, Kathleen Artificial Intelligence Computation and Language Human evaluation has been the gold standard for checking faithfulness in abstractive summarization. However, with a challenging source domain like narrative, multiple annotators can agree a summary is faithful, while missing details that are obvious errors only once pointed out. We therefore introduce a new dataset, STORYSUMM, comprising LLM summaries of short stories with localized faithfulness labels and error explanations. This benchmark is for evaluation methods, testing whether a given method can detect challenging inconsistencies. Using this dataset, we first show that any one human annotation protocol is likely to miss inconsistencies, and we advocate for pursuing a range of methods when establishing ground truth for a summarization dataset. We finally test recent automatic metrics and find that none of them achieve more than 70% balanced accuracy on this task, demonstrating that it is a challenging benchmark for future work in faithfulness evaluation. |
| title | STORYSUMM: Evaluating Faithfulness in Story Summarization |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2407.06501 |