Reading Subtext: Evaluating Large Language Models on Short Story Summarization with Writers
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911953115414528 |
|---|---|
| author | Subbiah, Melanie Zhang, Sean Chilton, Lydia B. McKeown, Kathleen |
| author_facet | Subbiah, Melanie Zhang, Sean Chilton, Lydia B. McKeown, Kathleen |
| contents | We evaluate recent Large Language Models (LLMs) on the challenging task of summarizing short stories, which can be lengthy, and include nuanced subtext or scrambled timelines. Importantly, we work directly with authors to ensure that the stories have not been shared online (and therefore are unseen by the models), and to obtain informed evaluations of summary quality using judgments from the authors themselves. Through quantitative and qualitative analysis grounded in narrative theory, we compare GPT-4, Claude-2.1, and LLama-2-70B. We find that all three models make faithfulness mistakes in over 50% of summaries and struggle with specificity and interpretation of difficult subtext. We additionally demonstrate that LLM ratings and other automatic metrics for summary quality do not correlate well with the quality ratings from the writers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_01061 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Reading Subtext: Evaluating Large Language Models on Short Story Summarization with Writers Subbiah, Melanie Zhang, Sean Chilton, Lydia B. McKeown, Kathleen Computation and Language We evaluate recent Large Language Models (LLMs) on the challenging task of summarizing short stories, which can be lengthy, and include nuanced subtext or scrambled timelines. Importantly, we work directly with authors to ensure that the stories have not been shared online (and therefore are unseen by the models), and to obtain informed evaluations of summary quality using judgments from the authors themselves. Through quantitative and qualitative analysis grounded in narrative theory, we compare GPT-4, Claude-2.1, and LLama-2-70B. We find that all three models make faithfulness mistakes in over 50% of summaries and struggle with specificity and interpretation of difficult subtext. We additionally demonstrate that LLM ratings and other automatic metrics for summary quality do not correlate well with the quality ratings from the writers. |
| title | Reading Subtext: Evaluating Large Language Models on Short Story Summarization with Writers |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2403.01061 |