Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2508.07273 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912530970968064 |
|---|---|
| author | Wang, Qiongqiong Sailor, Hardik B. Wong, Jeremy H. M. Liu, Tianchi Sun, Shuo Zhang, Wenyu Huzaifah, Muhammad Chen, Nancy Aw, Ai Ti |
| author_facet | Wang, Qiongqiong Sailor, Hardik B. Wong, Jeremy H. M. Liu, Tianchi Sun, Shuo Zhang, Wenyu Huzaifah, Muhammad Chen, Nancy Aw, Ai Ti |
| contents | Current large speech language models (Speech-LLMs) often exhibit limitations in empathetic reasoning, primarily due to the absence of training datasets that integrate both contextual content and paralinguistic cues. In this work, we propose two approaches to incorporate contextual paralinguistic information into model training: (1) an explicit method that provides paralinguistic metadata (e.g., emotion annotations) directly to the LLM, and (2) an implicit method that automatically generates novel training question-answer (QA) pairs using both categorical and dimensional emotion annotations alongside speech transcriptions. Our implicit method boosts performance (LLM-judged) by 38.41% on a human-annotated QA benchmark, reaching 46.02% when combined with the explicit approach, showing effectiveness in contextual paralinguistic understanding. We also validate the LLM judge by demonstrating its correlation with classification metrics, providing support for its reliability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_07273 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Incorporating Contextual Paralinguistic Understanding in Large Speech-Language Models Wang, Qiongqiong Sailor, Hardik B. Wong, Jeremy H. M. Liu, Tianchi Sun, Shuo Zhang, Wenyu Huzaifah, Muhammad Chen, Nancy Aw, Ai Ti Computation and Language Artificial Intelligence Audio and Speech Processing Current large speech language models (Speech-LLMs) often exhibit limitations in empathetic reasoning, primarily due to the absence of training datasets that integrate both contextual content and paralinguistic cues. In this work, we propose two approaches to incorporate contextual paralinguistic information into model training: (1) an explicit method that provides paralinguistic metadata (e.g., emotion annotations) directly to the LLM, and (2) an implicit method that automatically generates novel training question-answer (QA) pairs using both categorical and dimensional emotion annotations alongside speech transcriptions. Our implicit method boosts performance (LLM-judged) by 38.41% on a human-annotated QA benchmark, reaching 46.02% when combined with the explicit approach, showing effectiveness in contextual paralinguistic understanding. We also validate the LLM judge by demonstrating its correlation with classification metrics, providing support for its reliability. |
| title | Incorporating Contextual Paralinguistic Understanding in Large Speech-Language Models |
| topic | Computation and Language Artificial Intelligence Audio and Speech Processing |
| url | https://arxiv.org/abs/2508.07273 |