Beyond Silent Letters: Amplifying LLMs in Emotion Recognition with Vocal Nuances
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913623181361152 |
|---|---|
| author | Wu, Zehui Gong, Ziwei Ai, Lin Shi, Pengyuan Donbekci, Kaan Hirschberg, Julia |
| author_facet | Wu, Zehui Gong, Ziwei Ai, Lin Shi, Pengyuan Donbekci, Kaan Hirschberg, Julia |
| contents | Emotion recognition in speech is a challenging multimodal task that requires understanding both verbal content and vocal nuances. This paper introduces a novel approach to emotion detection using Large Language Models (LLMs), which have demonstrated exceptional capabilities in natural language understanding. To overcome the inherent limitation of LLMs in processing audio inputs, we propose SpeechCueLLM, a method that translates speech characteristics into natural language descriptions, allowing LLMs to perform multimodal emotion analysis via text prompts without any architectural changes. Our method is minimal yet impactful, outperforming baseline models that require structural modifications. We evaluate SpeechCueLLM on two datasets: IEMOCAP and MELD, showing significant improvements in emotion recognition accuracy, particularly for high-quality audio data. We also explore the effectiveness of various feature representations and fine-tuning strategies for different LLMs. Our experiments demonstrate that incorporating speech descriptions yields a more than 2% increase in the average weighted F1 score on IEMOCAP (from 70.111% to 72.596%). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_21315 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Beyond Silent Letters: Amplifying LLMs in Emotion Recognition with Vocal Nuances Wu, Zehui Gong, Ziwei Ai, Lin Shi, Pengyuan Donbekci, Kaan Hirschberg, Julia Computation and Language Artificial Intelligence Emotion recognition in speech is a challenging multimodal task that requires understanding both verbal content and vocal nuances. This paper introduces a novel approach to emotion detection using Large Language Models (LLMs), which have demonstrated exceptional capabilities in natural language understanding. To overcome the inherent limitation of LLMs in processing audio inputs, we propose SpeechCueLLM, a method that translates speech characteristics into natural language descriptions, allowing LLMs to perform multimodal emotion analysis via text prompts without any architectural changes. Our method is minimal yet impactful, outperforming baseline models that require structural modifications. We evaluate SpeechCueLLM on two datasets: IEMOCAP and MELD, showing significant improvements in emotion recognition accuracy, particularly for high-quality audio data. We also explore the effectiveness of various feature representations and fine-tuning strategies for different LLMs. Our experiments demonstrate that incorporating speech descriptions yields a more than 2% increase in the average weighted F1 score on IEMOCAP (from 70.111% to 72.596%). |
| title | Beyond Silent Letters: Amplifying LLMs in Emotion Recognition with Vocal Nuances |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2407.21315 |