Beyond Classification: Towards Speech Emotion Reasoning with Multitask AudioLLMs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912614360023040 |
|---|---|
| author | Zhang, Wenyu He, Yingxu Lin, Geyu Liu, Zhuohan Sun, Shuo Wang, Bin Zou, Xunlong Wong, Jeremy H. M. Wang, Qiongqiong Sailor, Hardik B. Chen, Nancy F. Aw, Ai Ti |
| author_facet | Zhang, Wenyu He, Yingxu Lin, Geyu Liu, Zhuohan Sun, Shuo Wang, Bin Zou, Xunlong Wong, Jeremy H. M. Wang, Qiongqiong Sailor, Hardik B. Chen, Nancy F. Aw, Ai Ti |
| contents | Audio Large Language Models (AudioLLMs) have achieved strong results in semantic tasks like speech recognition and translation, but remain limited in modeling paralinguistic cues such as emotion. Existing approaches often treat emotion understanding as a classification problem, offering little insight into the underlying rationale behind predictions. In this work, we explore emotion reasoning, a strategy that leverages the generative capabilities of AudioLLMs to enhance emotion recognition by producing semantically aligned, evidence-grounded explanations. To support this in multitask AudioLLMs, we introduce a unified framework combining reasoning-augmented data supervision, dual-encoder architecture, and task-alternating training. This approach enables AudioLLMs to effectively learn different tasks while incorporating emotional reasoning. Experiments on IEMOCAP and MELD show that our approach not only improves emotion prediction accuracy but also enhances the coherence and evidential grounding of the generated responses. Experiments on two out-of-domain datasets demonstrate the generalization capabilities of the resulting model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_06820 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Beyond Classification: Towards Speech Emotion Reasoning with Multitask AudioLLMs Zhang, Wenyu He, Yingxu Lin, Geyu Liu, Zhuohan Sun, Shuo Wang, Bin Zou, Xunlong Wong, Jeremy H. M. Wang, Qiongqiong Sailor, Hardik B. Chen, Nancy F. Aw, Ai Ti Computation and Language Sound Audio and Speech Processing Audio Large Language Models (AudioLLMs) have achieved strong results in semantic tasks like speech recognition and translation, but remain limited in modeling paralinguistic cues such as emotion. Existing approaches often treat emotion understanding as a classification problem, offering little insight into the underlying rationale behind predictions. In this work, we explore emotion reasoning, a strategy that leverages the generative capabilities of AudioLLMs to enhance emotion recognition by producing semantically aligned, evidence-grounded explanations. To support this in multitask AudioLLMs, we introduce a unified framework combining reasoning-augmented data supervision, dual-encoder architecture, and task-alternating training. This approach enables AudioLLMs to effectively learn different tasks while incorporating emotional reasoning. Experiments on IEMOCAP and MELD show that our approach not only improves emotion prediction accuracy but also enhances the coherence and evidential grounding of the generated responses. Experiments on two out-of-domain datasets demonstrate the generalization capabilities of the resulting model. |
| title | Beyond Classification: Towards Speech Emotion Reasoning with Multitask AudioLLMs |
| topic | Computation and Language Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2506.06820 |