Multilingual Fine-Grained News Headline Hallucination Detection
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909265187307520 |
|---|---|
| author | Shen, Jiaming Liu, Tianqi Liu, Jialu Qin, Zhen Pavagadhi, Jay Baumgartner, Simon Bendersky, Michael |
| author_facet | Shen, Jiaming Liu, Tianqi Liu, Jialu Qin, Zhen Pavagadhi, Jay Baumgartner, Simon Bendersky, Michael |
| contents | The popularity of automated news headline generation has surged with advancements in pre-trained language models. However, these models often suffer from the ``hallucination'' problem, where the generated headline is not fully supported by its source article. Efforts to address this issue have predominantly focused on English, using over-simplistic classification schemes that overlook nuanced hallucination types. In this study, we introduce the first multilingual, fine-grained news headline hallucination detection dataset that contains over 11 thousand pairs in 5 languages, each annotated with detailed hallucination types by experts. We conduct extensive experiments on this dataset under two settings. First, we implement several supervised fine-tuning approaches as preparatory solutions and demonstrate this dataset's challenges and utilities. Second, we test various large language models' in-context learning abilities and propose two novel techniques, language-dependent demonstration selection and coarse-to-fine prompting, to boost the few-shot hallucination detection performance in terms of the example-F1 metric. We release this dataset to foster further research in multilingual, fine-grained headline hallucination detection. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_15975 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Multilingual Fine-Grained News Headline Hallucination Detection Shen, Jiaming Liu, Tianqi Liu, Jialu Qin, Zhen Pavagadhi, Jay Baumgartner, Simon Bendersky, Michael Computation and Language The popularity of automated news headline generation has surged with advancements in pre-trained language models. However, these models often suffer from the ``hallucination'' problem, where the generated headline is not fully supported by its source article. Efforts to address this issue have predominantly focused on English, using over-simplistic classification schemes that overlook nuanced hallucination types. In this study, we introduce the first multilingual, fine-grained news headline hallucination detection dataset that contains over 11 thousand pairs in 5 languages, each annotated with detailed hallucination types by experts. We conduct extensive experiments on this dataset under two settings. First, we implement several supervised fine-tuning approaches as preparatory solutions and demonstrate this dataset's challenges and utilities. Second, we test various large language models' in-context learning abilities and propose two novel techniques, language-dependent demonstration selection and coarse-to-fine prompting, to boost the few-shot hallucination detection performance in terms of the example-F1 metric. We release this dataset to foster further research in multilingual, fine-grained headline hallucination detection. |
| title | Multilingual Fine-Grained News Headline Hallucination Detection |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2407.15975 |