Piculet: Specialized Models-Guided Hallucination Decrease for MultiModal Large Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917739623350272 |
|---|---|
| author | Wang, Kohou Liu, Xiang Liu, Zhaoxiang Wang, Kai Lian, Shiguo |
| author_facet | Wang, Kohou Liu, Xiang Liu, Zhaoxiang Wang, Kai Lian, Shiguo |
| contents | Multimodal Large Language Models (MLLMs) have made significant progress in bridging the gap between visual and language modalities. However, hallucinations in MLLMs, where the generated text does not align with image content, continue to be a major challenge. Existing methods for addressing hallucinations often rely on instruction-tuning, which requires retraining the model with specific data, which increases the cost of utilizing MLLMs further. In this paper, we introduce a novel training-free method, named Piculet, for enhancing the input representation of MLLMs. Piculet leverages multiple specialized models to extract descriptions of visual information from the input image and combine these descriptions with the original image and query as input to the MLLM. We evaluate our method both quantitively and qualitatively, and the results demonstrate that Piculet greatly decreases hallucinations of MLLMs. Our method can be easily extended to different MLLMs while being universal. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_01003 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Piculet: Specialized Models-Guided Hallucination Decrease for MultiModal Large Language Models Wang, Kohou Liu, Xiang Liu, Zhaoxiang Wang, Kai Lian, Shiguo Artificial Intelligence Multimodal Large Language Models (MLLMs) have made significant progress in bridging the gap between visual and language modalities. However, hallucinations in MLLMs, where the generated text does not align with image content, continue to be a major challenge. Existing methods for addressing hallucinations often rely on instruction-tuning, which requires retraining the model with specific data, which increases the cost of utilizing MLLMs further. In this paper, we introduce a novel training-free method, named Piculet, for enhancing the input representation of MLLMs. Piculet leverages multiple specialized models to extract descriptions of visual information from the input image and combine these descriptions with the original image and query as input to the MLLM. We evaluate our method both quantitively and qualitatively, and the results demonstrate that Piculet greatly decreases hallucinations of MLLMs. Our method can be easily extended to different MLLMs while being universal. |
| title | Piculet: Specialized Models-Guided Hallucination Decrease for MultiModal Large Language Models |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2408.01003 |