AILS-NTUA at SemEval-2024 Task 6: Efficient model tuning for hallucination detection and analysis
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911837141860352 |
|---|---|
| author | Grigoriadou, Natalia Lymperaiou, Maria Filandrianos, Giorgos Stamou, Giorgos |
| author_facet | Grigoriadou, Natalia Lymperaiou, Maria Filandrianos, Giorgos Stamou, Giorgos |
| contents | In this paper, we present our team's submissions for SemEval-2024 Task-6 - SHROOM, a Shared-task on Hallucinations and Related Observable Overgeneration Mistakes. The participants were asked to perform binary classification to identify cases of fluent overgeneration hallucinations. Our experimentation included fine-tuning a pre-trained model on hallucination detection and a Natural Language Inference (NLI) model. The most successful strategy involved creating an ensemble of these models, resulting in accuracy rates of 77.8% and 79.9% on model-agnostic and model-aware datasets respectively, outperforming the organizers' baseline and achieving notable results when contrasted with the top-performing results in the competition, which reported accuracies of 84.7% and 81.3% correspondingly. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_01210 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | AILS-NTUA at SemEval-2024 Task 6: Efficient model tuning for hallucination detection and analysis Grigoriadou, Natalia Lymperaiou, Maria Filandrianos, Giorgos Stamou, Giorgos Computation and Language In this paper, we present our team's submissions for SemEval-2024 Task-6 - SHROOM, a Shared-task on Hallucinations and Related Observable Overgeneration Mistakes. The participants were asked to perform binary classification to identify cases of fluent overgeneration hallucinations. Our experimentation included fine-tuning a pre-trained model on hallucination detection and a Natural Language Inference (NLI) model. The most successful strategy involved creating an ensemble of these models, resulting in accuracy rates of 77.8% and 79.9% on model-agnostic and model-aware datasets respectively, outperforming the organizers' baseline and achieving notable results when contrasted with the top-performing results in the competition, which reported accuracies of 84.7% and 81.3% correspondingly. |
| title | AILS-NTUA at SemEval-2024 Task 6: Efficient model tuning for hallucination detection and analysis |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2404.01210 |