Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2403.01988 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917741791805440 |
|---|---|
| author | Liu, Xuannan Li, Peipei Huang, Huaibo Li, Zekun Cui, Xing Liang, Jiahao Qin, Lixiong Deng, Weihong He, Zhaofeng |
| author_facet | Liu, Xuannan Li, Peipei Huang, Huaibo Li, Zekun Cui, Xing Liang, Jiahao Qin, Lixiong Deng, Weihong He, Zhaofeng |
| contents | The massive generation of multimodal fake news involving both text and images exhibits substantial distribution discrepancies, prompting the need for generalized detectors. However, the insulated nature of training restricts the capability of classical detectors to obtain open-world facts. While Large Vision-Language Models (LVLMs) have encoded rich world knowledge, they are not inherently tailored for combating fake news and struggle to comprehend local forgery details. In this paper, we propose FKA-Owl, a novel framework that leverages forgery-specific knowledge to augment LVLMs, enabling them to reason about manipulations effectively. The augmented forgery-specific knowledge includes semantic correlation between text and images, and artifact trace in image manipulation. To inject these two kinds of knowledge into the LVLM, we design two specialized modules to establish their representations, respectively. The encoded knowledge embeddings are then incorporated into LVLMs. Extensive experiments on the public benchmark demonstrate that FKA-Owl achieves superior cross-domain performance compared to previous methods. Code is publicly available at https://liuxuannan.github.io/FKA_Owl.github.io/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_01988 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | FKA-Owl: Advancing Multimodal Fake News Detection through Knowledge-Augmented LVLMs Liu, Xuannan Li, Peipei Huang, Huaibo Li, Zekun Cui, Xing Liang, Jiahao Qin, Lixiong Deng, Weihong He, Zhaofeng Computation and Language The massive generation of multimodal fake news involving both text and images exhibits substantial distribution discrepancies, prompting the need for generalized detectors. However, the insulated nature of training restricts the capability of classical detectors to obtain open-world facts. While Large Vision-Language Models (LVLMs) have encoded rich world knowledge, they are not inherently tailored for combating fake news and struggle to comprehend local forgery details. In this paper, we propose FKA-Owl, a novel framework that leverages forgery-specific knowledge to augment LVLMs, enabling them to reason about manipulations effectively. The augmented forgery-specific knowledge includes semantic correlation between text and images, and artifact trace in image manipulation. To inject these two kinds of knowledge into the LVLM, we design two specialized modules to establish their representations, respectively. The encoded knowledge embeddings are then incorporated into LVLMs. Extensive experiments on the public benchmark demonstrate that FKA-Owl achieves superior cross-domain performance compared to previous methods. Code is publicly available at https://liuxuannan.github.io/FKA_Owl.github.io/. |
| title | FKA-Owl: Advancing Multimodal Fake News Detection through Knowledge-Augmented LVLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2403.01988 |