Cross-Lingual Speech Emotion Recognition: Humans vs. Self-Supervised Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908343418748928 |
|---|---|
| author | Han, Zhichen Geng, Tianqi Feng, Hui Yuan, Jiahong Richmond, Korin Li, Yuanchao |
| author_facet | Han, Zhichen Geng, Tianqi Feng, Hui Yuan, Jiahong Richmond, Korin Li, Yuanchao |
| contents | Utilizing Self-Supervised Learning (SSL) models for Speech Emotion Recognition (SER) has proven effective, yet limited research has explored cross-lingual scenarios. This study presents a comparative analysis between human performance and SSL models, beginning with a layer-wise analysis and an exploration of parameter-efficient fine-tuning strategies in monolingual, cross-lingual, and transfer learning contexts. We further compare the SER ability of models and humans at both utterance- and segment-levels. Additionally, we investigate the impact of dialect on cross-lingual SER through human evaluation. Our findings reveal that models, with appropriate knowledge transfer, can adapt to the target language and achieve performance comparable to native speakers. We also demonstrate the significant effect of dialect on SER for individuals without prior linguistic and paralinguistic background. Moreover, both humans and models exhibit distinct behaviors across different emotions. These results offer new insights into the cross-lingual SER capabilities of SSL models, underscoring both their similarities to and differences from human emotion perception. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_16920 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Cross-Lingual Speech Emotion Recognition: Humans vs. Self-Supervised Models Han, Zhichen Geng, Tianqi Feng, Hui Yuan, Jiahong Richmond, Korin Li, Yuanchao Audio and Speech Processing Artificial Intelligence Computation and Language Human-Computer Interaction Sound Utilizing Self-Supervised Learning (SSL) models for Speech Emotion Recognition (SER) has proven effective, yet limited research has explored cross-lingual scenarios. This study presents a comparative analysis between human performance and SSL models, beginning with a layer-wise analysis and an exploration of parameter-efficient fine-tuning strategies in monolingual, cross-lingual, and transfer learning contexts. We further compare the SER ability of models and humans at both utterance- and segment-levels. Additionally, we investigate the impact of dialect on cross-lingual SER through human evaluation. Our findings reveal that models, with appropriate knowledge transfer, can adapt to the target language and achieve performance comparable to native speakers. We also demonstrate the significant effect of dialect on SER for individuals without prior linguistic and paralinguistic background. Moreover, both humans and models exhibit distinct behaviors across different emotions. These results offer new insights into the cross-lingual SER capabilities of SSL models, underscoring both their similarities to and differences from human emotion perception. |
| title | Cross-Lingual Speech Emotion Recognition: Humans vs. Self-Supervised Models |
| topic | Audio and Speech Processing Artificial Intelligence Computation and Language Human-Computer Interaction Sound |
| url | https://arxiv.org/abs/2409.16920 |