ForgerySleuth: Empowering Multimodal Large Language Models for Image Manipulation Detection
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914222364950528 |
|---|---|
| author | Sun, Zhihao Jiang, Haoran Chen, Haoran Cao, Yixin Qiu, Xipeng Wu, Zuxuan Jiang, Yu-Gang |
| author_facet | Sun, Zhihao Jiang, Haoran Chen, Haoran Cao, Yixin Qiu, Xipeng Wu, Zuxuan Jiang, Yu-Gang |
| contents | Multimodal large language models have unlocked new possibilities for various multimodal tasks. However, their potential in image manipulation detection remains unexplored. When directly applied to the IMD task, M-LLMs often produce reasoning texts that suffer from hallucinations and overthinking. To address this, we propose ForgerySleuth, which leverages M-LLMs to perform comprehensive clue fusion and generate segmentation outputs indicating specific regions that are tampered with. Moreover, we construct the ForgeryAnalysis dataset through the Chain-of-Clues prompt, which includes analysis and reasoning text to upgrade the image manipulation detection task. A data engine is also introduced to build a larger-scale dataset for the pre-training phase. Our extensive experiments demonstrate the effectiveness of ForgeryAnalysis and show that ForgerySleuth significantly outperforms existing methods in generalization, robustness, and explainability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_19466 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ForgerySleuth: Empowering Multimodal Large Language Models for Image Manipulation Detection Sun, Zhihao Jiang, Haoran Chen, Haoran Cao, Yixin Qiu, Xipeng Wu, Zuxuan Jiang, Yu-Gang Computer Vision and Pattern Recognition Machine Learning Multimodal large language models have unlocked new possibilities for various multimodal tasks. However, their potential in image manipulation detection remains unexplored. When directly applied to the IMD task, M-LLMs often produce reasoning texts that suffer from hallucinations and overthinking. To address this, we propose ForgerySleuth, which leverages M-LLMs to perform comprehensive clue fusion and generate segmentation outputs indicating specific regions that are tampered with. Moreover, we construct the ForgeryAnalysis dataset through the Chain-of-Clues prompt, which includes analysis and reasoning text to upgrade the image manipulation detection task. A data engine is also introduced to build a larger-scale dataset for the pre-training phase. Our extensive experiments demonstrate the effectiveness of ForgeryAnalysis and show that ForgerySleuth significantly outperforms existing methods in generalization, robustness, and explainability. |
| title | ForgerySleuth: Empowering Multimodal Large Language Models for Image Manipulation Detection |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2411.19466 |