Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2403.12244 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914719038701568 |
|---|---|
| author | Bhamidipati, Patanjali Malladi, Advaith Shrivastava, Manish Mamidi, Radhika |
| author_facet | Bhamidipati, Patanjali Malladi, Advaith Shrivastava, Manish Mamidi, Radhika |
| contents | In recent studies, the extensive utilization of large language models has underscored the importance of robust evaluation methodologies for assessing text generation quality and relevance to specific tasks. This has revealed a prevalent issue known as hallucination, an emergent condition in the model where generated text lacks faithfulness to the source and deviates from the evaluation criteria. In this study, we formally define hallucination and propose a framework for its quantitative detection in a zero-shot setting, leveraging our definition and the assumption that model outputs entail task and sample specific inputs. In detecting hallucinations, our solution achieves an accuracy of 0.78 in a model-aware setting and 0.61 in a model-agnostic setting. Notably, our solution maintains computational efficiency, requiring far less computational resources than other SOTA approaches, aligning with the trend towards lightweight and compressed models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_12244 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Zero-Shot Multi-task Hallucination Detection Bhamidipati, Patanjali Malladi, Advaith Shrivastava, Manish Mamidi, Radhika Computation and Language In recent studies, the extensive utilization of large language models has underscored the importance of robust evaluation methodologies for assessing text generation quality and relevance to specific tasks. This has revealed a prevalent issue known as hallucination, an emergent condition in the model where generated text lacks faithfulness to the source and deviates from the evaluation criteria. In this study, we formally define hallucination and propose a framework for its quantitative detection in a zero-shot setting, leveraging our definition and the assumption that model outputs entail task and sample specific inputs. In detecting hallucinations, our solution achieves an accuracy of 0.78 in a model-aware setting and 0.61 in a model-agnostic setting. Notably, our solution maintains computational efficiency, requiring far less computational resources than other SOTA approaches, aligning with the trend towards lightweight and compressed models. |
| title | Zero-Shot Multi-task Hallucination Detection |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2403.12244 |