Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.09071 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916911636283392 |
|---|---|
| author | Grenander, Matt Varia, Siddharth Czarnowska, Paula Vyas, Yogarshi Halder, Kishaloy Min, Bonan |
| author_facet | Grenander, Matt Varia, Siddharth Czarnowska, Paula Vyas, Yogarshi Halder, Kishaloy Min, Bonan |
| contents | Plan-guided summarization attempts to reduce hallucinations in small language models (SLMs) by grounding generated summaries to the source text, typically by targeting fine-grained details such as dates or named entities. In this work, we investigate whether plan-based approaches in SLMs improve summarization in long document, narrative tasks. Narrative texts' length and complexity often mean they are difficult to summarize faithfully. We analyze existing plan-guided solutions targeting fine-grained details, and also propose our own higher-level, narrative-based plan formulation. Our results show that neither approach significantly improves on a baseline without planning in either summary quality or faithfulness. Human evaluation reveals that while plan-guided approaches are often well grounded to their plan, plans are equally likely to contain hallucinations compared to summaries. As a result, the plan-guided summaries are just as unfaithful as those from models without planning. Our work serves as a cautionary tale to plan-guided approaches to summarization, especially for long, complex domains such as narrative texts. Code available at https://github.com/amazon-science/plan-guided-summarization |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_09071 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Exploration of Plan-Guided Summarization for Narrative Texts: the Case of Small Language Models Grenander, Matt Varia, Siddharth Czarnowska, Paula Vyas, Yogarshi Halder, Kishaloy Min, Bonan Computation and Language Plan-guided summarization attempts to reduce hallucinations in small language models (SLMs) by grounding generated summaries to the source text, typically by targeting fine-grained details such as dates or named entities. In this work, we investigate whether plan-based approaches in SLMs improve summarization in long document, narrative tasks. Narrative texts' length and complexity often mean they are difficult to summarize faithfully. We analyze existing plan-guided solutions targeting fine-grained details, and also propose our own higher-level, narrative-based plan formulation. Our results show that neither approach significantly improves on a baseline without planning in either summary quality or faithfulness. Human evaluation reveals that while plan-guided approaches are often well grounded to their plan, plans are equally likely to contain hallucinations compared to summaries. As a result, the plan-guided summaries are just as unfaithful as those from models without planning. Our work serves as a cautionary tale to plan-guided approaches to summarization, especially for long, complex domains such as narrative texts. Code available at https://github.com/amazon-science/plan-guided-summarization |
| title | Exploration of Plan-Guided Summarization for Narrative Texts: the Case of Small Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2504.09071 |