MetaCheckGPT -- A Multi-task Hallucination Detector Using LLM Uncertainty and Meta-models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916200828633088 |
|---|---|
| author | Mehta, Rahul Hoblitzell, Andrew O'Keefe, Jack Jang, Hyeju Varma, Vasudeva |
| author_facet | Mehta, Rahul Hoblitzell, Andrew O'Keefe, Jack Jang, Hyeju Varma, Vasudeva |
| contents | Hallucinations in large language models (LLMs) have recently become a significant problem. A recent effort in this direction is a shared task at Semeval 2024 Task 6, SHROOM, a Shared-task on Hallucinations and Related Observable Overgeneration Mistakes. This paper describes our winning solution ranked 1st and 2nd in the 2 sub-tasks of model agnostic and model aware tracks respectively. We propose a meta-regressor framework of LLMs for model evaluation and integration that achieves the highest scores on the leaderboard. We also experiment with various transformer-based models and black box methods like ChatGPT, Vectara, and others. In addition, we perform an error analysis comparing GPT4 against our best model which shows the limitations of the former. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_06948 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MetaCheckGPT -- A Multi-task Hallucination Detector Using LLM Uncertainty and Meta-models Mehta, Rahul Hoblitzell, Andrew O'Keefe, Jack Jang, Hyeju Varma, Vasudeva Computation and Language Artificial Intelligence 68T07, 68T50 I.2.7 Hallucinations in large language models (LLMs) have recently become a significant problem. A recent effort in this direction is a shared task at Semeval 2024 Task 6, SHROOM, a Shared-task on Hallucinations and Related Observable Overgeneration Mistakes. This paper describes our winning solution ranked 1st and 2nd in the 2 sub-tasks of model agnostic and model aware tracks respectively. We propose a meta-regressor framework of LLMs for model evaluation and integration that achieves the highest scores on the leaderboard. We also experiment with various transformer-based models and black box methods like ChatGPT, Vectara, and others. In addition, we perform an error analysis comparing GPT4 against our best model which shows the limitations of the former. |
| title | MetaCheckGPT -- A Multi-task Hallucination Detector Using LLM Uncertainty and Meta-models |
| topic | Computation and Language Artificial Intelligence 68T07, 68T50 I.2.7 |
| url | https://arxiv.org/abs/2404.06948 |