SpeechRole: A Large-Scale Dataset and Benchmark for Evaluating Speech Role-Playing Agents
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917361942003712 |
|---|---|
| author | Jiang, Changhao Sun, Jiajun Cao, Yifei Zhuang, Jiabao Che, Xinmeng Li, Hui Fan, Xiaoran Zhang, Ming Ye, Junjie Dou, Shihan Xi, Zhiheng Tong, Jingqi Wu, Yilong Fan, Baoyu Ji, Tao Gui, Tao Zhang, Qi Huang, Xuanjing |
| author_facet | Jiang, Changhao Sun, Jiajun Cao, Yifei Zhuang, Jiabao Che, Xinmeng Li, Hui Fan, Xiaoran Zhang, Ming Ye, Junjie Dou, Shihan Xi, Zhiheng Tong, Jingqi Wu, Yilong Fan, Baoyu Ji, Tao Gui, Tao Zhang, Qi Huang, Xuanjing |
| contents | Speech is essential for realistic role-playing, yet existing work on role-playing agents largely centers on text, leaving Speech Role-Playing Agents (SRPAs) underexplored and without systematic evaluation. We introduce SpeechRole, a unified framework for developing and assessing SRPAs. SpeechRole-Data contains 98 roles and 111k speech-to-speech conversations with rich timbre and prosodic variation, providing large-scale resources for training SRPAs. SpeechRole-Eval offers a multidimensional benchmark that directly evaluates generated speech, preserving paralinguistic cues and measuring interaction ability, speech expressiveness, and role-playing fidelity. Experiments show that end-to-end SRPAs such as GPT-4o Audio achieve strong fluency and naturalness, but remain limited in prosody consistency and emotion appropriateness. In contrast, current open-source end-to-end models exhibit substantial performance gaps across multiple evaluation dimensions. Cascaded and end-to-end systems achieve comparable results in interaction ability and role-playing fidelity, suggesting that these aspects are still largely influenced by the underlying text-based language models. We release all data, code, and evaluation tools at https://github.com/yuhui1038/SpeechRole. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_02013 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SpeechRole: A Large-Scale Dataset and Benchmark for Evaluating Speech Role-Playing Agents Jiang, Changhao Sun, Jiajun Cao, Yifei Zhuang, Jiabao Che, Xinmeng Li, Hui Fan, Xiaoran Zhang, Ming Ye, Junjie Dou, Shihan Xi, Zhiheng Tong, Jingqi Wu, Yilong Fan, Baoyu Ji, Tao Gui, Tao Zhang, Qi Huang, Xuanjing Computation and Language Speech is essential for realistic role-playing, yet existing work on role-playing agents largely centers on text, leaving Speech Role-Playing Agents (SRPAs) underexplored and without systematic evaluation. We introduce SpeechRole, a unified framework for developing and assessing SRPAs. SpeechRole-Data contains 98 roles and 111k speech-to-speech conversations with rich timbre and prosodic variation, providing large-scale resources for training SRPAs. SpeechRole-Eval offers a multidimensional benchmark that directly evaluates generated speech, preserving paralinguistic cues and measuring interaction ability, speech expressiveness, and role-playing fidelity. Experiments show that end-to-end SRPAs such as GPT-4o Audio achieve strong fluency and naturalness, but remain limited in prosody consistency and emotion appropriateness. In contrast, current open-source end-to-end models exhibit substantial performance gaps across multiple evaluation dimensions. Cascaded and end-to-end systems achieve comparable results in interaction ability and role-playing fidelity, suggesting that these aspects are still largely influenced by the underlying text-based language models. We release all data, code, and evaluation tools at https://github.com/yuhui1038/SpeechRole. |
| title | SpeechRole: A Large-Scale Dataset and Benchmark for Evaluating Speech Role-Playing Agents |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2508.02013 |