Ensembling Multiple Hallucination Detectors Trained on VLLM Internal Representations
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909976166924288 |
|---|---|
| author | Nakamizo, Yuto Miyazato, Ryuhei Tanabe, Hikaru Yamakura, Ryuta Hatanaka, Kiori |
| author_facet | Nakamizo, Yuto Miyazato, Ryuhei Tanabe, Hikaru Yamakura, Ryuta Hatanaka, Kiori |
| contents | This paper presents the 5th place solution by our team, y3h2, for the Meta CRAG-MM Challenge at KDD Cup 2025. The CRAG-MM benchmark is a visual question answering (VQA) dataset focused on factual questions about images, including egocentric images. The competition was contested based on VQA accuracy, as judged by an LLM-based automatic evaluator. Since incorrect answers result in negative scores, our strategy focused on reducing hallucinations from the internal representations of the VLM. Specifically, we trained logistic regression-based hallucination detection models using both the hidden_state and the outputs of specific attention heads. We then employed an ensemble of these models. As a result, while our method sacrificed some correct answers, it significantly reduced hallucinations and allowed us to place among the top entries on the final leaderboard. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_14330 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Ensembling Multiple Hallucination Detectors Trained on VLLM Internal Representations Nakamizo, Yuto Miyazato, Ryuhei Tanabe, Hikaru Yamakura, Ryuta Hatanaka, Kiori Information Retrieval This paper presents the 5th place solution by our team, y3h2, for the Meta CRAG-MM Challenge at KDD Cup 2025. The CRAG-MM benchmark is a visual question answering (VQA) dataset focused on factual questions about images, including egocentric images. The competition was contested based on VQA accuracy, as judged by an LLM-based automatic evaluator. Since incorrect answers result in negative scores, our strategy focused on reducing hallucinations from the internal representations of the VLM. Specifically, we trained logistic regression-based hallucination detection models using both the hidden_state and the outputs of specific attention heads. We then employed an ensemble of these models. As a result, while our method sacrificed some correct answers, it significantly reduced hallucinations and allowed us to place among the top entries on the final leaderboard. |
| title | Ensembling Multiple Hallucination Detectors Trained on VLLM Internal Representations |
| topic | Information Retrieval |
| url | https://arxiv.org/abs/2510.14330 |