Enabling Auditory Large Language Models for Automatic Speech Quality Evaluation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909560539709440 |
|---|---|
| author | Wang, Siyin Yu, Wenyi Yang, Yudong Tang, Changli Li, Yixuan Zhuang, Jimin Chen, Xianzhao Tian, Xiaohai Zhang, Jun Sun, Guangzhi Lu, Lu Wang, Yuxuan Zhang, Chao |
| author_facet | Wang, Siyin Yu, Wenyi Yang, Yudong Tang, Changli Li, Yixuan Zhuang, Jimin Chen, Xianzhao Tian, Xiaohai Zhang, Jun Sun, Guangzhi Lu, Lu Wang, Yuxuan Zhang, Chao |
| contents | Speech quality assessment typically requires evaluating audio from multiple aspects, such as mean opinion score (MOS) and speaker similarity (SIM) \etc., which can be challenging to cover using one small model designed for a single task. In this paper, we propose leveraging recently introduced auditory large language models (LLMs) for automatic speech quality assessment. By employing task-specific prompts, auditory LLMs are finetuned to predict MOS, SIM and A/B testing results, which are commonly used for evaluating text-to-speech systems. Additionally, the finetuned auditory LLM is able to generate natural language descriptions assessing aspects like noisiness, distortion, discontinuity, and overall quality, providing more interpretable outputs. Extensive experiments have been performed on the NISQA, BVCC, SOMOS and VoxSim speech quality datasets, using open-source auditory LLMs such as SALMONN, Qwen-Audio, and Qwen2-Audio. For the natural language descriptions task, a commercial model Google Gemini 1.5 Pro is also evaluated. The results demonstrate that auditory LLMs achieve competitive performance compared to state-of-the-art task-specific small models in predicting MOS and SIM, while also delivering promising results in A/B testing and natural language descriptions. Our data processing scripts and finetuned model checkpoints can be found at https://github.com/bytedance/SALMONN. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_16644 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Enabling Auditory Large Language Models for Automatic Speech Quality Evaluation Wang, Siyin Yu, Wenyi Yang, Yudong Tang, Changli Li, Yixuan Zhuang, Jimin Chen, Xianzhao Tian, Xiaohai Zhang, Jun Sun, Guangzhi Lu, Lu Wang, Yuxuan Zhang, Chao Audio and Speech Processing Computation and Language Sound Speech quality assessment typically requires evaluating audio from multiple aspects, such as mean opinion score (MOS) and speaker similarity (SIM) \etc., which can be challenging to cover using one small model designed for a single task. In this paper, we propose leveraging recently introduced auditory large language models (LLMs) for automatic speech quality assessment. By employing task-specific prompts, auditory LLMs are finetuned to predict MOS, SIM and A/B testing results, which are commonly used for evaluating text-to-speech systems. Additionally, the finetuned auditory LLM is able to generate natural language descriptions assessing aspects like noisiness, distortion, discontinuity, and overall quality, providing more interpretable outputs. Extensive experiments have been performed on the NISQA, BVCC, SOMOS and VoxSim speech quality datasets, using open-source auditory LLMs such as SALMONN, Qwen-Audio, and Qwen2-Audio. For the natural language descriptions task, a commercial model Google Gemini 1.5 Pro is also evaluated. The results demonstrate that auditory LLMs achieve competitive performance compared to state-of-the-art task-specific small models in predicting MOS and SIM, while also delivering promising results in A/B testing and natural language descriptions. Our data processing scripts and finetuned model checkpoints can be found at https://github.com/bytedance/SALMONN. |
| title | Enabling Auditory Large Language Models for Automatic Speech Quality Evaluation |
| topic | Audio and Speech Processing Computation and Language Sound |
| url | https://arxiv.org/abs/2409.16644 |