OSUM-EChat: Enhancing End-to-End Empathetic Spoken Chatbot via Understanding-Driven Spoken Dialogue
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911135571116032 |
|---|---|
| author | Geng, Xuelong Shao, Qijie Xue, Hongfei Wang, Shuiyuan Xie, Hanke Guo, Zhao Zhao, Yi Li, Guojian Tian, Wenjie Wang, Chengyou Zhao, Zhixian Xia, Kangxiang Zhang, Ziyu Lin, Zhennan Zuo, Tianlun Shao, Mingchen Cao, Yuang Ma, Guobin Li, Longhao Dai, Yuhang Gao, Dehui Guo, Dake Xie, Lei |
| author_facet | Geng, Xuelong Shao, Qijie Xue, Hongfei Wang, Shuiyuan Xie, Hanke Guo, Zhao Zhao, Yi Li, Guojian Tian, Wenjie Wang, Chengyou Zhao, Zhixian Xia, Kangxiang Zhang, Ziyu Lin, Zhennan Zuo, Tianlun Shao, Mingchen Cao, Yuang Ma, Guobin Li, Longhao Dai, Yuhang Gao, Dehui Guo, Dake Xie, Lei |
| contents | Empathy is crucial in enabling natural interactions within spoken dialogue systems, allowing machines to recognize and respond appropriately to paralinguistic cues such as age, gender, and emotion. Recent advancements in end-to-end speech language models, which unify speech understanding and generation, provide promising solutions. However, several challenges persist, including an over-reliance on large-scale dialogue datasets, insufficient extraction of paralinguistic cues vital for conveying empathy, and the lack of empathy-specific datasets and evaluation frameworks. To address these issues, we introduce OSUM-EChat, an open-source, end-to-end spoken dialogue system designed to enhance empathetic interactions, particularly in resource-limited settings. OSUM-EChat introduces two key innovations: (1) a three-stage understanding-driven spoken dialogue training strategy that extends the capabilities of a large speech understanding model to spoken dialogue tasks, and (2) a linguistic-paralinguistic dual thinking mechanism that integrates paralinguistic understanding through a chain of thought with dialogue generation, enabling the system to produce more empathetic responses. This approach reduces reliance on large-scale dialogue datasets while maintaining high-quality empathetic interactions. Additionally, we introduce the EChat-200K dataset, a rich corpus of empathetic speech-to-speech dialogues, and the EChat-eval benchmark, a comprehensive framework for evaluating the empathetic capabilities of dialogue systems. Experimental results demonstrate that OSUM-EChat outperforms end-to-end spoken dialogue models regarding empathetic responsiveness, validating its effectiveness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_09600 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | OSUM-EChat: Enhancing End-to-End Empathetic Spoken Chatbot via Understanding-Driven Spoken Dialogue Geng, Xuelong Shao, Qijie Xue, Hongfei Wang, Shuiyuan Xie, Hanke Guo, Zhao Zhao, Yi Li, Guojian Tian, Wenjie Wang, Chengyou Zhao, Zhixian Xia, Kangxiang Zhang, Ziyu Lin, Zhennan Zuo, Tianlun Shao, Mingchen Cao, Yuang Ma, Guobin Li, Longhao Dai, Yuhang Gao, Dehui Guo, Dake Xie, Lei Sound Empathy is crucial in enabling natural interactions within spoken dialogue systems, allowing machines to recognize and respond appropriately to paralinguistic cues such as age, gender, and emotion. Recent advancements in end-to-end speech language models, which unify speech understanding and generation, provide promising solutions. However, several challenges persist, including an over-reliance on large-scale dialogue datasets, insufficient extraction of paralinguistic cues vital for conveying empathy, and the lack of empathy-specific datasets and evaluation frameworks. To address these issues, we introduce OSUM-EChat, an open-source, end-to-end spoken dialogue system designed to enhance empathetic interactions, particularly in resource-limited settings. OSUM-EChat introduces two key innovations: (1) a three-stage understanding-driven spoken dialogue training strategy that extends the capabilities of a large speech understanding model to spoken dialogue tasks, and (2) a linguistic-paralinguistic dual thinking mechanism that integrates paralinguistic understanding through a chain of thought with dialogue generation, enabling the system to produce more empathetic responses. This approach reduces reliance on large-scale dialogue datasets while maintaining high-quality empathetic interactions. Additionally, we introduce the EChat-200K dataset, a rich corpus of empathetic speech-to-speech dialogues, and the EChat-eval benchmark, a comprehensive framework for evaluating the empathetic capabilities of dialogue systems. Experimental results demonstrate that OSUM-EChat outperforms end-to-end spoken dialogue models regarding empathetic responsiveness, validating its effectiveness. |
| title | OSUM-EChat: Enhancing End-to-End Empathetic Spoken Chatbot via Understanding-Driven Spoken Dialogue |
| topic | Sound |
| url | https://arxiv.org/abs/2508.09600 |