Towards Source Attribution of Singing Voice Deepfake with Multimodal Foundation Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912412767092736 |
|---|---|
| author | Phukan, Orchid Chetia Girish Akhtar, Mohd Mujtaba Behera, Swarup Ranjan Mallick, Priyabrata Reddy, Pailla Balakrishna Buduru, Arun Balaji Sharma, Rajesh |
| author_facet | Phukan, Orchid Chetia Girish Akhtar, Mohd Mujtaba Behera, Swarup Ranjan Mallick, Priyabrata Reddy, Pailla Balakrishna Buduru, Arun Balaji Sharma, Rajesh |
| contents | In this work, we introduce the task of singing voice deepfake source attribution (SVDSA). We hypothesize that multimodal foundation models (MMFMs) such as ImageBind, LanguageBind will be most effective for SVDSA as they are better equipped for capturing subtle source-specific characteristics-such as unique timbre, pitch manipulation, or synthesis artifacts of each singing voice deepfake source due to their cross-modality pre-training. Our experiments with MMFMs, speech foundation models and music foundation models verify the hypothesis that MMFMs are the most effective for SVDSA. Furthermore, inspired from related research, we also explore fusion of foundation models (FMs) for improved SVDSA. To this end, we propose a novel framework, COFFE which employs Chernoff Distance as novel loss function for effective fusion of FMs. Through COFFE with the symphony of MMFMs, we attain the topmost performance in comparison to all the individual FMs and baseline fusion methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_03364 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Towards Source Attribution of Singing Voice Deepfake with Multimodal Foundation Models Phukan, Orchid Chetia Girish Akhtar, Mohd Mujtaba Behera, Swarup Ranjan Mallick, Priyabrata Reddy, Pailla Balakrishna Buduru, Arun Balaji Sharma, Rajesh Audio and Speech Processing Multimedia Sound In this work, we introduce the task of singing voice deepfake source attribution (SVDSA). We hypothesize that multimodal foundation models (MMFMs) such as ImageBind, LanguageBind will be most effective for SVDSA as they are better equipped for capturing subtle source-specific characteristics-such as unique timbre, pitch manipulation, or synthesis artifacts of each singing voice deepfake source due to their cross-modality pre-training. Our experiments with MMFMs, speech foundation models and music foundation models verify the hypothesis that MMFMs are the most effective for SVDSA. Furthermore, inspired from related research, we also explore fusion of foundation models (FMs) for improved SVDSA. To this end, we propose a novel framework, COFFE which employs Chernoff Distance as novel loss function for effective fusion of FMs. Through COFFE with the symphony of MMFMs, we attain the topmost performance in comparison to all the individual FMs and baseline fusion methods. |
| title | Towards Source Attribution of Singing Voice Deepfake with Multimodal Foundation Models |
| topic | Audio and Speech Processing Multimedia Sound |
| url | https://arxiv.org/abs/2506.03364 |