Correction with Backtracking Reduces Hallucination in Summarization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916380534636544 |
|---|---|
| author | Liu, Zhenzhen Wan, Chao Kishore, Varsha Zhou, Jin Peng Chen, Minmin Weinberger, Kilian Q. |
| author_facet | Liu, Zhenzhen Wan, Chao Kishore, Varsha Zhou, Jin Peng Chen, Minmin Weinberger, Kilian Q. |
| contents | Abstractive summarization aims at generating natural language summaries of a source document that are succinct while preserving the important elements. Despite recent advances, neural text summarization models are known to be susceptible to hallucinating (or more correctly confabulating), that is to produce summaries with details that are not grounded in the source document. In this paper, we introduce a simple yet efficient technique, CoBa, to reduce hallucination in abstractive summarization. The approach is based on two steps: hallucination detection and mitigation. We show that the former can be achieved through measuring simple statistics about conditional word probabilities and distance to context words. Further, we demonstrate that straight-forward backtracking is surprisingly effective at mitigation. We thoroughly evaluate the proposed method with prior art on three benchmark datasets for text summarization. The results show that CoBa is effective and efficient in reducing hallucination, and offers great adaptability and flexibility. Code can be found at https://github.com/zhenzhel/CoBa. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_16176 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Correction with Backtracking Reduces Hallucination in Summarization Liu, Zhenzhen Wan, Chao Kishore, Varsha Zhou, Jin Peng Chen, Minmin Weinberger, Kilian Q. Computation and Language Artificial Intelligence Abstractive summarization aims at generating natural language summaries of a source document that are succinct while preserving the important elements. Despite recent advances, neural text summarization models are known to be susceptible to hallucinating (or more correctly confabulating), that is to produce summaries with details that are not grounded in the source document. In this paper, we introduce a simple yet efficient technique, CoBa, to reduce hallucination in abstractive summarization. The approach is based on two steps: hallucination detection and mitigation. We show that the former can be achieved through measuring simple statistics about conditional word probabilities and distance to context words. Further, we demonstrate that straight-forward backtracking is surprisingly effective at mitigation. We thoroughly evaluate the proposed method with prior art on three benchmark datasets for text summarization. The results show that CoBa is effective and efficient in reducing hallucination, and offers great adaptability and flexibility. Code can be found at https://github.com/zhenzhel/CoBa. |
| title | Correction with Backtracking Reduces Hallucination in Summarization |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2310.16176 |