FactCHD: Benchmarking Fact-Conflicting Hallucination Detection
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914812324216832 |
|---|---|
| author | Chen, Xiang Song, Duanzheng Gui, Honghao Wang, Chenxi Zhang, Ningyu Jiang, Yong Huang, Fei Lv, Chengfei Zhang, Dan Chen, Huajun |
| author_facet | Chen, Xiang Song, Duanzheng Gui, Honghao Wang, Chenxi Zhang, Ningyu Jiang, Yong Huang, Fei Lv, Chengfei Zhang, Dan Chen, Huajun |
| contents | Despite their impressive generative capabilities, LLMs are hindered by fact-conflicting hallucinations in real-world applications. The accurate identification of hallucinations in texts generated by LLMs, especially in complex inferential scenarios, is a relatively unexplored area. To address this gap, we present FactCHD, a dedicated benchmark designed for the detection of fact-conflicting hallucinations from LLMs. FactCHD features a diverse dataset that spans various factuality patterns, including vanilla, multi-hop, comparison, and set operation. A distinctive element of FactCHD is its integration of fact-based evidence chains, significantly enhancing the depth of evaluating the detectors' explanations. Experiments on different LLMs expose the shortcomings of current approaches in detecting factual errors accurately. Furthermore, we introduce Truth-Triangulator that synthesizes reflective considerations by tool-enhanced ChatGPT and LoRA-tuning based on Llama2, aiming to yield more credible detection through the amalgamation of predictive results and evidence. The benchmark dataset is available at https://github.com/zjunlp/FactCHD. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_12086 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | FactCHD: Benchmarking Fact-Conflicting Hallucination Detection Chen, Xiang Song, Duanzheng Gui, Honghao Wang, Chenxi Zhang, Ningyu Jiang, Yong Huang, Fei Lv, Chengfei Zhang, Dan Chen, Huajun Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition Information Retrieval Machine Learning Despite their impressive generative capabilities, LLMs are hindered by fact-conflicting hallucinations in real-world applications. The accurate identification of hallucinations in texts generated by LLMs, especially in complex inferential scenarios, is a relatively unexplored area. To address this gap, we present FactCHD, a dedicated benchmark designed for the detection of fact-conflicting hallucinations from LLMs. FactCHD features a diverse dataset that spans various factuality patterns, including vanilla, multi-hop, comparison, and set operation. A distinctive element of FactCHD is its integration of fact-based evidence chains, significantly enhancing the depth of evaluating the detectors' explanations. Experiments on different LLMs expose the shortcomings of current approaches in detecting factual errors accurately. Furthermore, we introduce Truth-Triangulator that synthesizes reflective considerations by tool-enhanced ChatGPT and LoRA-tuning based on Llama2, aiming to yield more credible detection through the amalgamation of predictive results and evidence. The benchmark dataset is available at https://github.com/zjunlp/FactCHD. |
| title | FactCHD: Benchmarking Fact-Conflicting Hallucination Detection |
| topic | Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition Information Retrieval Machine Learning |
| url | https://arxiv.org/abs/2310.12086 |