Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.05177 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918171909292032 |
|---|---|
| author | Wang, Chen Peng, Tianyu Yang, Wen Bai, Yinan Wang, Guangfu Lin, Jun Jia, Lanpeng Wu, Lingxiang Wang, Jinqiao Zong, Chengqing Zhang, Jiajun |
| author_facet | Wang, Chen Peng, Tianyu Yang, Wen Bai, Yinan Wang, Guangfu Lin, Jun Jia, Lanpeng Wu, Lingxiang Wang, Jinqiao Zong, Chengqing Zhang, Jiajun |
| contents | Empathetic interaction is a cornerstone of human-machine communication, due to the need for understanding speech enriched with paralinguistic cues and generating emotional and expressive responses. However, the most powerful empathetic LSLMs are increasingly closed off, leaving the crucial details about the architecture, data and development opaque to researchers. Given the critical need for transparent research into the LSLMs and empathetic behavior, we present OpenS2S, a fully open-source, transparent and end-to-end LSLM designed to enable empathetic speech interactions. Based on our empathetic speech-to-text model BLSP-Emo, OpenS2S further employs a streaming interleaved decoding architecture to achieve low-latency speech generation. To facilitate end-to-end training, OpenS2S incorporates an automated data construction pipeline that synthesizes diverse, high-quality empathetic speech dialogues at low cost. By leveraging large language models to generate empathetic content and controllable text-to-speech systems to introduce speaker and emotional variation, we construct a scalable training corpus with rich paralinguistic diversity and minimal human supervision. We release the fully open-source OpenS2S model, including the dataset, model weights, pre-training and fine-tuning codes, to empower the broader research community and accelerate innovation in empathetic speech systems. The project webpage can be accessed at https://casia-lm.github.io/OpenS2S |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_05177 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Model Wang, Chen Peng, Tianyu Yang, Wen Bai, Yinan Wang, Guangfu Lin, Jun Jia, Lanpeng Wu, Lingxiang Wang, Jinqiao Zong, Chengqing Zhang, Jiajun Computation and Language Artificial Intelligence Sound Audio and Speech Processing Empathetic interaction is a cornerstone of human-machine communication, due to the need for understanding speech enriched with paralinguistic cues and generating emotional and expressive responses. However, the most powerful empathetic LSLMs are increasingly closed off, leaving the crucial details about the architecture, data and development opaque to researchers. Given the critical need for transparent research into the LSLMs and empathetic behavior, we present OpenS2S, a fully open-source, transparent and end-to-end LSLM designed to enable empathetic speech interactions. Based on our empathetic speech-to-text model BLSP-Emo, OpenS2S further employs a streaming interleaved decoding architecture to achieve low-latency speech generation. To facilitate end-to-end training, OpenS2S incorporates an automated data construction pipeline that synthesizes diverse, high-quality empathetic speech dialogues at low cost. By leveraging large language models to generate empathetic content and controllable text-to-speech systems to introduce speaker and emotional variation, we construct a scalable training corpus with rich paralinguistic diversity and minimal human supervision. We release the fully open-source OpenS2S model, including the dataset, model weights, pre-training and fine-tuning codes, to empower the broader research community and accelerate innovation in empathetic speech systems. The project webpage can be accessed at https://casia-lm.github.io/OpenS2S |
| title | OpenS2S: Advancing Fully Open-Source End-to-End Empathetic Large Speech Language Model |
| topic | Computation and Language Artificial Intelligence Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2507.05177 |