Circumventing shortcuts in audio-visual deepfake detection datasets with unsupervised learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913865639395328 |
|---|---|
| author | Smeu, Stefan Boldisor, Dragos-Alexandru Oneata, Dan Oneata, Elisabeta |
| author_facet | Smeu, Stefan Boldisor, Dragos-Alexandru Oneata, Dan Oneata, Elisabeta |
| contents | Good datasets are essential for developing and benchmarking any machine learning system. Their importance is even more extreme for safety critical applications such as deepfake detection - the focus of this paper. Here we reveal that two of the most widely used audio-video deepfake datasets suffer from a previously unidentified spurious feature: the leading silence. Fake videos start with a very brief moment of silence and based on this feature alone, we can separate the real and fake samples almost perfectly. As such, previous audio-only and audio-video models exploit the presence of silence in the fake videos and consequently perform worse when the leading silence is removed. To circumvent latching on such unwanted artifact and possibly other unrevealed ones we propose a shift from supervised to unsupervised learning by training models exclusively on real data. We show that by aligning self-supervised audio-video representations we remove the risk of relying on dataset-specific biases and improve robustness in deepfake detection. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_00175 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Circumventing shortcuts in audio-visual deepfake detection datasets with unsupervised learning Smeu, Stefan Boldisor, Dragos-Alexandru Oneata, Dan Oneata, Elisabeta Computer Vision and Pattern Recognition Machine Learning Sound Audio and Speech Processing Image and Video Processing Good datasets are essential for developing and benchmarking any machine learning system. Their importance is even more extreme for safety critical applications such as deepfake detection - the focus of this paper. Here we reveal that two of the most widely used audio-video deepfake datasets suffer from a previously unidentified spurious feature: the leading silence. Fake videos start with a very brief moment of silence and based on this feature alone, we can separate the real and fake samples almost perfectly. As such, previous audio-only and audio-video models exploit the presence of silence in the fake videos and consequently perform worse when the leading silence is removed. To circumvent latching on such unwanted artifact and possibly other unrevealed ones we propose a shift from supervised to unsupervised learning by training models exclusively on real data. We show that by aligning self-supervised audio-video representations we remove the risk of relying on dataset-specific biases and improve robustness in deepfake detection. |
| title | Circumventing shortcuts in audio-visual deepfake detection datasets with unsupervised learning |
| topic | Computer Vision and Pattern Recognition Machine Learning Sound Audio and Speech Processing Image and Video Processing |
| url | https://arxiv.org/abs/2412.00175 |