Identifying Factual Inconsistencies in Summaries: Grounding LLM Inference via Task Taxonomy
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910632941453312 |
|---|---|
| author | Xu, Liyan Su, Zhenlin Yu, Mo Xu, Jin Choi, Jinho D. Zhou, Jie Liu, Fei |
| author_facet | Xu, Liyan Su, Zhenlin Yu, Mo Xu, Jin Choi, Jinho D. Zhou, Jie Liu, Fei |
| contents | Factual inconsistencies pose a significant hurdle for the faithful summarization by generative models. While a major direction to enhance inconsistency detection is to derive stronger Natural Language Inference (NLI) models, we propose an orthogonal aspect that underscores the importance of incorporating task-specific taxonomy into the inference. To this end, we consolidate key error types of inconsistent facts in summaries, and incorporate them to facilitate both the zero-shot and supervised paradigms of LLMs. Extensive experiments on ten datasets of five distinct domains suggest that, zero-shot LLM inference could benefit from the explicit solution space depicted by the error type taxonomy, and achieves state-of-the-art performance overall, surpassing specialized non-LLM baselines, as well as recent LLM baselines. We further distill models that fuse the taxonomy into parameters through our designed prompt completions and supervised training strategies, efficiently substituting state-of-the-art zero-shot inference with much larger LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_12821 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Identifying Factual Inconsistencies in Summaries: Grounding LLM Inference via Task Taxonomy Xu, Liyan Su, Zhenlin Yu, Mo Xu, Jin Choi, Jinho D. Zhou, Jie Liu, Fei Computation and Language Machine Learning Factual inconsistencies pose a significant hurdle for the faithful summarization by generative models. While a major direction to enhance inconsistency detection is to derive stronger Natural Language Inference (NLI) models, we propose an orthogonal aspect that underscores the importance of incorporating task-specific taxonomy into the inference. To this end, we consolidate key error types of inconsistent facts in summaries, and incorporate them to facilitate both the zero-shot and supervised paradigms of LLMs. Extensive experiments on ten datasets of five distinct domains suggest that, zero-shot LLM inference could benefit from the explicit solution space depicted by the error type taxonomy, and achieves state-of-the-art performance overall, surpassing specialized non-LLM baselines, as well as recent LLM baselines. We further distill models that fuse the taxonomy into parameters through our designed prompt completions and supervised training strategies, efficiently substituting state-of-the-art zero-shot inference with much larger LLMs. |
| title | Identifying Factual Inconsistencies in Summaries: Grounding LLM Inference via Task Taxonomy |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2402.12821 |