Mitigating Object Hallucinations in MLLMs via Multi-Frequency Perturbations
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916656669786112 |
|---|---|
| author | Li, Shuo Sun, Jiajun Zheng, Guodong Fan, Xiaoran Shen, Yujiong Lu, Yi Xi, Zhiheng Yang, Yuming Tan, Wenming Ji, Tao Gui, Tao Zhang, Qi Huang, Xuanjing |
| author_facet | Li, Shuo Sun, Jiajun Zheng, Guodong Fan, Xiaoran Shen, Yujiong Lu, Yi Xi, Zhiheng Yang, Yuming Tan, Wenming Ji, Tao Gui, Tao Zhang, Qi Huang, Xuanjing |
| contents | Recently, multimodal large language models (MLLMs) have demonstrated remarkable performance in visual-language tasks. However, the authenticity of the responses generated by MLLMs is often compromised by object hallucinations. We identify that a key cause of these hallucinations is the model's over-susceptibility to specific image frequency features in detecting objects. In this paper, we introduce Multi-Frequency Perturbations (MFP), a simple, cost-effective, and pluggable method that leverages both low-frequency and high-frequency features of images to perturb visual feature representations and explicitly suppress redundant frequency-domain features during inference, thereby mitigating hallucinations. Experimental results demonstrate that our method significantly mitigates object hallucinations across various model architectures. Furthermore, as a training-time method, MFP can be combined with inference-time methods to achieve state-of-the-art performance on the CHAIR benchmark. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_14895 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Mitigating Object Hallucinations in MLLMs via Multi-Frequency Perturbations Li, Shuo Sun, Jiajun Zheng, Guodong Fan, Xiaoran Shen, Yujiong Lu, Yi Xi, Zhiheng Yang, Yuming Tan, Wenming Ji, Tao Gui, Tao Zhang, Qi Huang, Xuanjing Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Recently, multimodal large language models (MLLMs) have demonstrated remarkable performance in visual-language tasks. However, the authenticity of the responses generated by MLLMs is often compromised by object hallucinations. We identify that a key cause of these hallucinations is the model's over-susceptibility to specific image frequency features in detecting objects. In this paper, we introduce Multi-Frequency Perturbations (MFP), a simple, cost-effective, and pluggable method that leverages both low-frequency and high-frequency features of images to perturb visual feature representations and explicitly suppress redundant frequency-domain features during inference, thereby mitigating hallucinations. Experimental results demonstrate that our method significantly mitigates object hallucinations across various model architectures. Furthermore, as a training-time method, MFP can be combined with inference-time methods to achieve state-of-the-art performance on the CHAIR benchmark. |
| title | Mitigating Object Hallucinations in MLLMs via Multi-Frequency Perturbations |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2503.14895 |