Cross-Modality and Within-Modality Regularization for Audio-Visual DeepFake Detection
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929206749822976 |
|---|---|
| author | Zou, Heqing Shen, Meng Hu, Yuchen Chen, Chen Chng, Eng Siong Rajan, Deepu |
| author_facet | Zou, Heqing Shen, Meng Hu, Yuchen Chen, Chen Chng, Eng Siong Rajan, Deepu |
| contents | Audio-visual deepfake detection scrutinizes manipulations in public video using complementary multimodal cues. Current methods, which train on fused multimodal data for multimodal targets face challenges due to uncertainties and inconsistencies in learned representations caused by independent modality manipulations in deepfake videos. To address this, we propose cross-modality and within-modality regularization to preserve modality distinctions during multimodal representation learning. Our approach includes an audio-visual transformer module for modality correspondence and a cross-modality regularization module to align paired audio-visual signals, preserving modality distinctions. Simultaneously, a within-modality regularization module refines unimodal representations with modality-specific targets to retain modal-specific details. Experimental results on the public audio-visual dataset, FakeAVCeleb, demonstrate the effectiveness and competitiveness of our approach. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_05746 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Cross-Modality and Within-Modality Regularization for Audio-Visual DeepFake Detection Zou, Heqing Shen, Meng Hu, Yuchen Chen, Chen Chng, Eng Siong Rajan, Deepu Multimedia Audio-visual deepfake detection scrutinizes manipulations in public video using complementary multimodal cues. Current methods, which train on fused multimodal data for multimodal targets face challenges due to uncertainties and inconsistencies in learned representations caused by independent modality manipulations in deepfake videos. To address this, we propose cross-modality and within-modality regularization to preserve modality distinctions during multimodal representation learning. Our approach includes an audio-visual transformer module for modality correspondence and a cross-modality regularization module to align paired audio-visual signals, preserving modality distinctions. Simultaneously, a within-modality regularization module refines unimodal representations with modality-specific targets to retain modal-specific details. Experimental results on the public audio-visual dataset, FakeAVCeleb, demonstrate the effectiveness and competitiveness of our approach. |
| title | Cross-Modality and Within-Modality Regularization for Audio-Visual DeepFake Detection |
| topic | Multimedia |
| url | https://arxiv.org/abs/2401.05746 |