From Talking to Singing: A New Challenge for Audio-Visual Deepfake Detection
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913166768734208 |
|---|---|
| author | Liu, Ke Wei, Jiwei Zhang, Wenyu Zhou, Shuchang Chai, Ruikun Dai, Yutao Zhang, Chaoning Yang, Yang |
| author_facet | Liu, Ke Wei, Jiwei Zhang, Wenyu Zhou, Shuchang Chai, Ruikun Dai, Yutao Zhang, Chaoning Yang, Yang |
| contents | With rapid advances in audio-visual generative models, reliable forgery detection becomes increasingly critical. Existing methods for audio-visual deepfake detection typically rely on cross-modal inconsistencies. In singing, rhythmic vocalization weakens this coupling and introduces a nontrivial domain shift, substantially degrading detection performance. We construct the Singing Head DeepFake (SHDF) dataset using rhythm-aware generative models to fill the gap in singing benchmarks. To cope with cross-scenario domain shifts, we propose a Text-guided Audio-Visual Forgery Detection (T-AVFD) framework that generalizes across both talking and singing scenarios. T-AVFD comprises a facial authenticity pattern learner and a multi-modal differential weight learning module. The pattern learner aligns facial features with multi-granularity textual descriptions to learn generalizable authenticity patterns. The weight learning module preserves intrinsic audio-visual consistency and adaptively integrates it with authenticity patterns via differential weighting. Extensive experiments on multiple talking head deepfake datasets and SHDF show consistent improvements over existing baselines and strong robustness under diverse perturbations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_27944 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | From Talking to Singing: A New Challenge for Audio-Visual Deepfake Detection Liu, Ke Wei, Jiwei Zhang, Wenyu Zhou, Shuchang Chai, Ruikun Dai, Yutao Zhang, Chaoning Yang, Yang Artificial Intelligence Multimedia Sound With rapid advances in audio-visual generative models, reliable forgery detection becomes increasingly critical. Existing methods for audio-visual deepfake detection typically rely on cross-modal inconsistencies. In singing, rhythmic vocalization weakens this coupling and introduces a nontrivial domain shift, substantially degrading detection performance. We construct the Singing Head DeepFake (SHDF) dataset using rhythm-aware generative models to fill the gap in singing benchmarks. To cope with cross-scenario domain shifts, we propose a Text-guided Audio-Visual Forgery Detection (T-AVFD) framework that generalizes across both talking and singing scenarios. T-AVFD comprises a facial authenticity pattern learner and a multi-modal differential weight learning module. The pattern learner aligns facial features with multi-granularity textual descriptions to learn generalizable authenticity patterns. The weight learning module preserves intrinsic audio-visual consistency and adaptively integrates it with authenticity patterns via differential weighting. Extensive experiments on multiple talking head deepfake datasets and SHDF show consistent improvements over existing baselines and strong robustness under diverse perturbations. |
| title | From Talking to Singing: A New Challenge for Audio-Visual Deepfake Detection |
| topic | Artificial Intelligence Multimedia Sound |
| url | https://arxiv.org/abs/2605.27944 |