Towards Long Context Hallucination Detection
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915262499913728 |
|---|---|
| author | Liu, Siyi Halder, Kishaloy Qi, Zheng Xiao, Wei Pappas, Nikolaos Htut, Phu Mon John, Neha Anna Benajiba, Yassine Roth, Dan |
| author_facet | Liu, Siyi Halder, Kishaloy Qi, Zheng Xiao, Wei Pappas, Nikolaos Htut, Phu Mon John, Neha Anna Benajiba, Yassine Roth, Dan |
| contents | Large Language Models (LLMs) have demonstrated remarkable performance across various tasks. However, they are prone to contextual hallucination, generating information that is either unsubstantiated or contradictory to the given context. Although many studies have investigated contextual hallucinations in LLMs, addressing them in long-context inputs remains an open problem. In this work, we take an initial step toward solving this problem by constructing a dataset specifically designed for long-context hallucination detection. Furthermore, we propose a novel architecture that enables pre-trained encoder models, such as BERT, to process long contexts and effectively detect contextual hallucinations through a decomposition and aggregation mechanism. Our experimental results show that the proposed architecture significantly outperforms previous models of similar size as well as LLM-based models across various metrics, while providing substantially faster inference. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_19457 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Towards Long Context Hallucination Detection Liu, Siyi Halder, Kishaloy Qi, Zheng Xiao, Wei Pappas, Nikolaos Htut, Phu Mon John, Neha Anna Benajiba, Yassine Roth, Dan Computation and Language Artificial Intelligence Large Language Models (LLMs) have demonstrated remarkable performance across various tasks. However, they are prone to contextual hallucination, generating information that is either unsubstantiated or contradictory to the given context. Although many studies have investigated contextual hallucinations in LLMs, addressing them in long-context inputs remains an open problem. In this work, we take an initial step toward solving this problem by constructing a dataset specifically designed for long-context hallucination detection. Furthermore, we propose a novel architecture that enables pre-trained encoder models, such as BERT, to process long contexts and effectively detect contextual hallucinations through a decomposition and aggregation mechanism. Our experimental results show that the proposed architecture significantly outperforms previous models of similar size as well as LLM-based models across various metrics, while providing substantially faster inference. |
| title | Towards Long Context Hallucination Detection |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2504.19457 |