Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.16193 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916958378655744 |
|---|---|
| author | Akhtar, Mohd Mujtaba Girish Phukan, Orchid Chetia Behera, Swarup Ranjan Reddy, Pailla Balakrishna Nayak, Ananda Chandra Nayak, Sanjib Kumar Buduru, Arun Balaji |
| author_facet | Akhtar, Mohd Mujtaba Girish Phukan, Orchid Chetia Behera, Swarup Ranjan Reddy, Pailla Balakrishna Nayak, Ananda Chandra Nayak, Sanjib Kumar Buduru, Arun Balaji |
| contents | In this work, we investigate multimodal foundation models (MFMs) for EmoFake detection (EFD) and hypothesize that they will outperform audio foundation models (AFMs). MFMs due to their cross-modal pre-training, learns emotional patterns from multiple modalities, while AFMs rely only on audio. As such, MFMs can better recognize unnatural emotional shifts and inconsistencies in manipulated audio, making them more effective at distinguishing real from fake emotional expressions. To validate our hypothesis, we conduct a comprehensive comparative analysis of state-of-the-art (SOTA) MFMs (e.g. LanguageBind) alongside AFMs (e.g. WavLM). Our experiments confirm that MFMs surpass AFMs for EFD. Beyond individual foundation models (FMs) performance, we explore FMs fusion, motivated by findings in related research areas such synthetic speech detection and speech emotion recognition. To this end, we propose SCAR, a novel framework for effective fusion. SCAR introduces a nested cross-attention mechanism, where representations from FMs interact at two stages sequentially to refine information exchange. Additionally, a self-attention refinement module further enhances feature representations by reinforcing important cross-FM cues while suppressing noise. Through SCAR with synergistic fusion of MFMs, we achieve SOTA performance, surpassing both standalone FMs and conventional fusion approaches and previous works on EFD. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_16193 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Are Multimodal Foundation Models All That Is Needed for Emofake Detection? Akhtar, Mohd Mujtaba Girish Phukan, Orchid Chetia Behera, Swarup Ranjan Reddy, Pailla Balakrishna Nayak, Ananda Chandra Nayak, Sanjib Kumar Buduru, Arun Balaji Audio and Speech Processing In this work, we investigate multimodal foundation models (MFMs) for EmoFake detection (EFD) and hypothesize that they will outperform audio foundation models (AFMs). MFMs due to their cross-modal pre-training, learns emotional patterns from multiple modalities, while AFMs rely only on audio. As such, MFMs can better recognize unnatural emotional shifts and inconsistencies in manipulated audio, making them more effective at distinguishing real from fake emotional expressions. To validate our hypothesis, we conduct a comprehensive comparative analysis of state-of-the-art (SOTA) MFMs (e.g. LanguageBind) alongside AFMs (e.g. WavLM). Our experiments confirm that MFMs surpass AFMs for EFD. Beyond individual foundation models (FMs) performance, we explore FMs fusion, motivated by findings in related research areas such synthetic speech detection and speech emotion recognition. To this end, we propose SCAR, a novel framework for effective fusion. SCAR introduces a nested cross-attention mechanism, where representations from FMs interact at two stages sequentially to refine information exchange. Additionally, a self-attention refinement module further enhances feature representations by reinforcing important cross-FM cues while suppressing noise. Through SCAR with synergistic fusion of MFMs, we achieve SOTA performance, surpassing both standalone FMs and conventional fusion approaches and previous works on EFD. |
| title | Are Multimodal Foundation Models All That Is Needed for Emofake Detection? |
| topic | Audio and Speech Processing |
| url | https://arxiv.org/abs/2509.16193 |