Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.23449 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915578301644800 |
|---|---|
| author | Li, Fanxiao Wu, Jiaying He, Canyuan Zhou, Wei |
| author_facet | Li, Fanxiao Wu, Jiaying He, Canyuan Zhou, Wei |
| contents | Multimodal large language models (MLLMs) have demonstrated impressive capabilities in visual reasoning and text generation. While previous studies have explored the application of MLLM for detecting out-of-context (OOC) misinformation, our empirical analysis reveals two persisting challenges of this paradigm. Evaluating the representative GPT-4o model on direct reasoning and evidence augmented reasoning, results indicate that MLLM struggle to capture the deeper relationships-specifically, cases in which the image and text are not directly connected but are associated through underlying semantic links. Moreover, noise in the evidence further impairs detection accuracy. To address these challenges, we propose CMIE, a novel OOC misinformation detection framework that incorporates a Coexistence Relationship Generation (CRG) strategy and an Association Scoring (AS) mechanism. CMIE identifies the underlying coexistence relationships between images and text, and selectively utilizes relevant evidence to enhance misinformation detection. Experimental results demonstrate that our approach outperforms existing methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_23449 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CMIE: Combining MLLM Insights with External Evidence for Explainable Out-of-Context Misinformation Detection Li, Fanxiao Wu, Jiaying He, Canyuan Zhou, Wei Multimedia Computer Vision and Pattern Recognition Information Retrieval Multimodal large language models (MLLMs) have demonstrated impressive capabilities in visual reasoning and text generation. While previous studies have explored the application of MLLM for detecting out-of-context (OOC) misinformation, our empirical analysis reveals two persisting challenges of this paradigm. Evaluating the representative GPT-4o model on direct reasoning and evidence augmented reasoning, results indicate that MLLM struggle to capture the deeper relationships-specifically, cases in which the image and text are not directly connected but are associated through underlying semantic links. Moreover, noise in the evidence further impairs detection accuracy. To address these challenges, we propose CMIE, a novel OOC misinformation detection framework that incorporates a Coexistence Relationship Generation (CRG) strategy and an Association Scoring (AS) mechanism. CMIE identifies the underlying coexistence relationships between images and text, and selectively utilizes relevant evidence to enhance misinformation detection. Experimental results demonstrate that our approach outperforms existing methods. |
| title | CMIE: Combining MLLM Insights with External Evidence for Explainable Out-of-Context Misinformation Detection |
| topic | Multimedia Computer Vision and Pattern Recognition Information Retrieval |
| url | https://arxiv.org/abs/2505.23449 |