Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2412.04939 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912777704046592 |
|---|---|
| author | Wang, Zehao Liu, Xinpeng Zhang, Yudonglin Wu, Xiaoqian Fang, Zhou Fang, Yifan Pu, Junfu Lu, Cewu Li, Yong-Lu |
| author_facet | Wang, Zehao Liu, Xinpeng Zhang, Yudonglin Wu, Xiaoqian Fang, Zhou Fang, Yifan Pu, Junfu Lu, Cewu Li, Yong-Lu |
| contents | Multimodal Large Language Models (MLLMs) have garnered significant attention recently and demonstrate outstanding capabilities in various tasks such as OCR, VQA, captioning, $\textit{etc}$. However, hallucination remains a persistent issue. While numerous methods have been proposed to mitigate hallucinations, achieving notable improvements, these methods primarily focus on mitigating hallucinations about $\textbf{object/noun-related}$ concepts. Verb concepts, crucial for understanding human actions, have been largely overlooked. In this paper, to the best of our knowledge, we are the $\textbf{first}$ to investigate the $\textbf{verb hallucination}$ phenomenon of MLLMs from various perspectives. Our findings reveal that most state-of-the-art MLLMs suffer from severe verb hallucination. To assess the effectiveness of existing mitigation methods for object concept hallucination on verb hallucination, we evaluated these methods and found that they do not effectively address verb hallucination. To address this issue, we propose a novel rich verb knowledge-based tuning method to mitigate verb hallucination. The experiment results demonstrate that our method significantly reduces hallucinations related to verbs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_04939 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Verb Mirage: Unveiling and Assessing Verb Concept Hallucinations in Multimodal Large Language Models Wang, Zehao Liu, Xinpeng Zhang, Yudonglin Wu, Xiaoqian Fang, Zhou Fang, Yifan Pu, Junfu Lu, Cewu Li, Yong-Lu Computer Vision and Pattern Recognition Multimodal Large Language Models (MLLMs) have garnered significant attention recently and demonstrate outstanding capabilities in various tasks such as OCR, VQA, captioning, $\textit{etc}$. However, hallucination remains a persistent issue. While numerous methods have been proposed to mitigate hallucinations, achieving notable improvements, these methods primarily focus on mitigating hallucinations about $\textbf{object/noun-related}$ concepts. Verb concepts, crucial for understanding human actions, have been largely overlooked. In this paper, to the best of our knowledge, we are the $\textbf{first}$ to investigate the $\textbf{verb hallucination}$ phenomenon of MLLMs from various perspectives. Our findings reveal that most state-of-the-art MLLMs suffer from severe verb hallucination. To assess the effectiveness of existing mitigation methods for object concept hallucination on verb hallucination, we evaluated these methods and found that they do not effectively address verb hallucination. To address this issue, we propose a novel rich verb knowledge-based tuning method to mitigate verb hallucination. The experiment results demonstrate that our method significantly reduces hallucinations related to verbs. |
| title | Verb Mirage: Unveiling and Assessing Verb Concept Hallucinations in Multimodal Large Language Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.04939 |