Aligning Paralinguistic Understanding and Generation in Speech LLMs via Multi-Task Reinforcement Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914399984287744 |
|---|---|
| author | Chen, Jingxiang Kim, Minseok Leem, Seong-Gyun Huang, Yin Rungta, Rashi Ouyang, Zhicheng Wu, Haibin Appini, Surya Teja Bansal, Ankur Bai, Yang Liu, Yue Metze, Florian Aly, Ahmed A Kumar, Anuj Rastrow, Ariya Lin, Zhaojiang |
| author_facet | Chen, Jingxiang Kim, Minseok Leem, Seong-Gyun Huang, Yin Rungta, Rashi Ouyang, Zhicheng Wu, Haibin Appini, Surya Teja Bansal, Ankur Bai, Yang Liu, Yue Metze, Florian Aly, Ahmed A Kumar, Anuj Rastrow, Ariya Lin, Zhaojiang |
| contents | Speech large language models (LLMs) observe paralinguistic cues such as prosody, emotion, and non-verbal sounds--crucial for intent understanding. However, leveraging these cues faces challenges: limited training data, annotation difficulty, and models exploiting lexical shortcuts over paralinguistic signals. We propose multi-task reinforcement learning (RL) with chain-of-thought prompting that elicits explicit affective reasoning. To address data scarcity, we introduce a paralinguistics-aware speech LLM (PALLM) that jointly optimizes sentiment classification from audio and paralinguistics-aware response generation via a two-stage pipeline. Experiments demonstrate that our approach improves paralinguistics understanding over both supervised baselines and strong proprietary models (Gemini-2.5-Pro, GPT-4o-audio) by 8-12% on Expresso, IEMOCAP, and RAVDESS. The results show that modeling paralinguistic reasoning with multi-task RL is crucial for building emotionally intelligent speech LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_15981 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Aligning Paralinguistic Understanding and Generation in Speech LLMs via Multi-Task Reinforcement Learning Chen, Jingxiang Kim, Minseok Leem, Seong-Gyun Huang, Yin Rungta, Rashi Ouyang, Zhicheng Wu, Haibin Appini, Surya Teja Bansal, Ankur Bai, Yang Liu, Yue Metze, Florian Aly, Ahmed A Kumar, Anuj Rastrow, Ariya Lin, Zhaojiang Computation and Language Artificial Intelligence Speech large language models (LLMs) observe paralinguistic cues such as prosody, emotion, and non-verbal sounds--crucial for intent understanding. However, leveraging these cues faces challenges: limited training data, annotation difficulty, and models exploiting lexical shortcuts over paralinguistic signals. We propose multi-task reinforcement learning (RL) with chain-of-thought prompting that elicits explicit affective reasoning. To address data scarcity, we introduce a paralinguistics-aware speech LLM (PALLM) that jointly optimizes sentiment classification from audio and paralinguistics-aware response generation via a two-stage pipeline. Experiments demonstrate that our approach improves paralinguistics understanding over both supervised baselines and strong proprietary models (Gemini-2.5-Pro, GPT-4o-audio) by 8-12% on Expresso, IEMOCAP, and RAVDESS. The results show that modeling paralinguistic reasoning with multi-task RL is crucial for building emotionally intelligent speech LLMs. |
| title | Aligning Paralinguistic Understanding and Generation in Speech LLMs via Multi-Task Reinforcement Learning |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2603.15981 |