Deepfake Word Detection by Next-token Prediction using Fine-tuned Whisper
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917300286783488 |
|---|---|
| author | Tran, Hoan My Wang, Xin Ge, Wanying Liu, Xuechen Yamagishi, Junichi |
| author_facet | Tran, Hoan My Wang, Xin Ge, Wanying Liu, Xuechen Yamagishi, Junichi |
| contents | Deepfake speech utterances can be forged by replacing one or more words in a bona fide utterance with semantically different words synthesized with speech-generative models. While a dedicated synthetic word detector could be developed, we developed a cost-effective method that fine-tunes a pre-trained Whisper model to detect synthetic words while transcribing the input utterance via next-token prediction. We further investigate using partially vocoded utterances as the fine-tuning data, thus reducing the cost of data collection. Our experiments demonstrate that, on in-domain test data, the fine-tuned Whisper yields low synthetic-word detection error rates and transcription error rates. On out-of-domain test data with synthetic words produced with unseen speech-generative models, the fine-tuned Whisper remains on par with a dedicated ResNet-based detection model; however, the overall performance degradation calls for strategies to improve its generalization capability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_22658 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Deepfake Word Detection by Next-token Prediction using Fine-tuned Whisper Tran, Hoan My Wang, Xin Ge, Wanying Liu, Xuechen Yamagishi, Junichi Audio and Speech Processing Computation and Language Deepfake speech utterances can be forged by replacing one or more words in a bona fide utterance with semantically different words synthesized with speech-generative models. While a dedicated synthetic word detector could be developed, we developed a cost-effective method that fine-tunes a pre-trained Whisper model to detect synthetic words while transcribing the input utterance via next-token prediction. We further investigate using partially vocoded utterances as the fine-tuning data, thus reducing the cost of data collection. Our experiments demonstrate that, on in-domain test data, the fine-tuned Whisper yields low synthetic-word detection error rates and transcription error rates. On out-of-domain test data with synthetic words produced with unseen speech-generative models, the fine-tuned Whisper remains on par with a dedicated ResNet-based detection model; however, the overall performance degradation calls for strategies to improve its generalization capability. |
| title | Deepfake Word Detection by Next-token Prediction using Fine-tuned Whisper |
| topic | Audio and Speech Processing Computation and Language |
| url | https://arxiv.org/abs/2602.22658 |