Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.12474 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918188604719104 |
|---|---|
| author | Zhou, Weixiao Zhu, Junnan Li, Gengyao Cheng, Xianfu Liang, Xinnian Zhai, Feifei Li, Zhoujun |
| author_facet | Zhou, Weixiao Zhu, Junnan Li, Gengyao Cheng, Xianfu Liang, Xinnian Zhai, Feifei Li, Zhoujun |
| contents | Traditional dialogue summarization primarily focuses on dialogue content, assuming it comprises adequate information for a clear summary. However, this assumption often fails for discussions grounded in shared background, where participants frequently omit context and use implicit references. This results in summaries that are confusing to readers unfamiliar with the background. To address this, we introduce Knowledge-Grounded Discussion Summarization (KGDS), a novel task that produces a supplementary background summary for context and a clear opinion summary with clarified references. To facilitate research, we construct the first KGDS benchmark, featuring news-discussion pairs and expert-created multi-granularity gold annotations for evaluating sub-summaries. We also propose a novel hierarchical evaluation framework with fine-grained and interpretable metrics. Our extensive evaluation of 12 advanced large language models (LLMs) reveals that KGDS remains a significant challenge. The models frequently miss key facts and retain irrelevant ones in background summarization, and often fail to resolve implicit references in opinion summary integration. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_12474 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | What Are They Talking About? A Benchmark of Knowledge-Grounded Discussion Summarization Zhou, Weixiao Zhu, Junnan Li, Gengyao Cheng, Xianfu Liang, Xinnian Zhai, Feifei Li, Zhoujun Computation and Language Traditional dialogue summarization primarily focuses on dialogue content, assuming it comprises adequate information for a clear summary. However, this assumption often fails for discussions grounded in shared background, where participants frequently omit context and use implicit references. This results in summaries that are confusing to readers unfamiliar with the background. To address this, we introduce Knowledge-Grounded Discussion Summarization (KGDS), a novel task that produces a supplementary background summary for context and a clear opinion summary with clarified references. To facilitate research, we construct the first KGDS benchmark, featuring news-discussion pairs and expert-created multi-granularity gold annotations for evaluating sub-summaries. We also propose a novel hierarchical evaluation framework with fine-grained and interpretable metrics. Our extensive evaluation of 12 advanced large language models (LLMs) reveals that KGDS remains a significant challenge. The models frequently miss key facts and retain irrelevant ones in background summarization, and often fail to resolve implicit references in opinion summary integration. |
| title | What Are They Talking About? A Benchmark of Knowledge-Grounded Discussion Summarization |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.12474 |