VocalBench-zh: Decomposing and Benchmarking the Speech Conversational Abilities in Mandarin Context
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909905121705984 |
|---|---|
| author | Liu, Heyang Cheng, Ziyang Wang, Yuhao Liu, Hongcheng Li, Yiqi Wu, Ronghua Gu, Qunshan Wang, Yanfeng Wang, Yu |
| author_facet | Liu, Heyang Cheng, Ziyang Wang, Yuhao Liu, Hongcheng Li, Yiqi Wu, Ronghua Gu, Qunshan Wang, Yanfeng Wang, Yu |
| contents | The development of multi-modal large language models (LLMs) leads to intelligent approaches capable of speech interactions. As one of the most widely spoken languages globally, Mandarin is supported by most models to enhance their applicability and reach. However, the scarcity of comprehensive speech-to-speech (S2S) benchmarks in Mandarin contexts impedes systematic evaluation for developers and hinders fair model comparison for users. In this work, we propose VocalBench-zh, an ability-level divided evaluation suite adapted to Mandarin context consisting of 10 well-crafted subsets and over 10K high-quality instances, covering 12 user-oriented characters. The evaluation experiment on 14 mainstream models reveals the common challenges for current routes, and highlights the need for new insights into next-generation speech interactive systems. The evaluation codes and datasets will be available at https://github.com/SJTU-OmniAgent/VocalBench-zh. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_08230 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | VocalBench-zh: Decomposing and Benchmarking the Speech Conversational Abilities in Mandarin Context Liu, Heyang Cheng, Ziyang Wang, Yuhao Liu, Hongcheng Li, Yiqi Wu, Ronghua Gu, Qunshan Wang, Yanfeng Wang, Yu Computation and Language The development of multi-modal large language models (LLMs) leads to intelligent approaches capable of speech interactions. As one of the most widely spoken languages globally, Mandarin is supported by most models to enhance their applicability and reach. However, the scarcity of comprehensive speech-to-speech (S2S) benchmarks in Mandarin contexts impedes systematic evaluation for developers and hinders fair model comparison for users. In this work, we propose VocalBench-zh, an ability-level divided evaluation suite adapted to Mandarin context consisting of 10 well-crafted subsets and over 10K high-quality instances, covering 12 user-oriented characters. The evaluation experiment on 14 mainstream models reveals the common challenges for current routes, and highlights the need for new insights into next-generation speech interactive systems. The evaluation codes and datasets will be available at https://github.com/SJTU-OmniAgent/VocalBench-zh. |
| title | VocalBench-zh: Decomposing and Benchmarking the Speech Conversational Abilities in Mandarin Context |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2511.08230 |