SpeechForensics: Audio-Visual Speech Representation Learning for Face Forgery Detection
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915444676362240 |
|---|---|
| author | Liang, Yachao Yu, Min Li, Gang Jiang, Jianguo Li, Boquan Yu, Feng Zhang, Ning Meng, Xiang Huang, Weiqing |
| author_facet | Liang, Yachao Yu, Min Li, Gang Jiang, Jianguo Li, Boquan Yu, Feng Zhang, Ning Meng, Xiang Huang, Weiqing |
| contents | Detection of face forgery videos remains a formidable challenge in the field of digital forensics, especially the generalization to unseen datasets and common perturbations. In this paper, we tackle this issue by leveraging the synergy between audio and visual speech elements, embarking on a novel approach through audio-visual speech representation learning. Our work is motivated by the finding that audio signals, enriched with speech content, can provide precise information effectively reflecting facial movements. To this end, we first learn precise audio-visual speech representations on real videos via a self-supervised masked prediction task, which encodes both local and global semantic information simultaneously. Then, the derived model is directly transferred to the forgery detection task. Extensive experiments demonstrate that our method outperforms the state-of-the-art methods in terms of cross-dataset generalization and robustness, without the participation of any fake video in model training. Code is available at https://github.com/Eleven4AI/SpeechForensics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_09913 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SpeechForensics: Audio-Visual Speech Representation Learning for Face Forgery Detection Liang, Yachao Yu, Min Li, Gang Jiang, Jianguo Li, Boquan Yu, Feng Zhang, Ning Meng, Xiang Huang, Weiqing Computer Vision and Pattern Recognition Detection of face forgery videos remains a formidable challenge in the field of digital forensics, especially the generalization to unseen datasets and common perturbations. In this paper, we tackle this issue by leveraging the synergy between audio and visual speech elements, embarking on a novel approach through audio-visual speech representation learning. Our work is motivated by the finding that audio signals, enriched with speech content, can provide precise information effectively reflecting facial movements. To this end, we first learn precise audio-visual speech representations on real videos via a self-supervised masked prediction task, which encodes both local and global semantic information simultaneously. Then, the derived model is directly transferred to the forgery detection task. Extensive experiments demonstrate that our method outperforms the state-of-the-art methods in terms of cross-dataset generalization and robustness, without the participation of any fake video in model training. Code is available at https://github.com/Eleven4AI/SpeechForensics. |
| title | SpeechForensics: Audio-Visual Speech Representation Learning for Face Forgery Detection |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2508.09913 |