Improving Faithfulness of Abstractive Summarization by Controlling Confounding Effect of Irrelevant Sentences
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2022
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914645502066688 |
|---|---|
| author | Ghoshal, Asish Einolghozati, Arash Arun, Ankit Li, Haoran Yu, Lili Gor, Vera Mehdad, Yashar Yih, Scott Wen-tau Celikyilmaz, Asli |
| author_facet | Ghoshal, Asish Einolghozati, Arash Arun, Ankit Li, Haoran Yu, Lili Gor, Vera Mehdad, Yashar Yih, Scott Wen-tau Celikyilmaz, Asli |
| contents | Lack of factual correctness is an issue that still plagues state-of-the-art summarization systems despite their impressive progress on generating seemingly fluent summaries. In this paper, we show that factual inconsistency can be caused by irrelevant parts of the input text, which act as confounders. To that end, we leverage information-theoretic measures of causal effects to quantify the amount of confounding and precisely quantify how they affect the summarization performance. Based on insights derived from our theoretical results, we design a simple multi-task model to control such confounding by leveraging human-annotated relevant sentences when available. Crucially, we give a principled characterization of data distributions where such confounding can be large thereby necessitating the use of human annotated relevant sentences to generate factual summaries. Our approach improves faithfulness scores by 20\% over strong baselines on AnswerSumm \citep{fabbri2021answersumm}, a conversation summarization dataset where lack of faithfulness is a significant issue due to the subjective nature of the task. Our best method achieves the highest faithfulness score while also achieving state-of-the-art results on standard metrics like ROUGE and METEOR. We corroborate these improvements through human evaluation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2212_09726 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | Improving Faithfulness of Abstractive Summarization by Controlling Confounding Effect of Irrelevant Sentences Ghoshal, Asish Einolghozati, Arash Arun, Ankit Li, Haoran Yu, Lili Gor, Vera Mehdad, Yashar Yih, Scott Wen-tau Celikyilmaz, Asli Computation and Language Artificial Intelligence Machine Learning Lack of factual correctness is an issue that still plagues state-of-the-art summarization systems despite their impressive progress on generating seemingly fluent summaries. In this paper, we show that factual inconsistency can be caused by irrelevant parts of the input text, which act as confounders. To that end, we leverage information-theoretic measures of causal effects to quantify the amount of confounding and precisely quantify how they affect the summarization performance. Based on insights derived from our theoretical results, we design a simple multi-task model to control such confounding by leveraging human-annotated relevant sentences when available. Crucially, we give a principled characterization of data distributions where such confounding can be large thereby necessitating the use of human annotated relevant sentences to generate factual summaries. Our approach improves faithfulness scores by 20\% over strong baselines on AnswerSumm \citep{fabbri2021answersumm}, a conversation summarization dataset where lack of faithfulness is a significant issue due to the subjective nature of the task. Our best method achieves the highest faithfulness score while also achieving state-of-the-art results on standard metrics like ROUGE and METEOR. We corroborate these improvements through human evaluation. |
| title | Improving Faithfulness of Abstractive Summarization by Controlling Confounding Effect of Irrelevant Sentences |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2212.09726 |