AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911061196668928 |
|---|---|
| author | Manakul, Potsawee Gan, Woody Haosheng Ryan, Michael J. Khan, Ali Sartaz Sirichotedumrong, Warit Pipatanakul, Kunat Held, William Yang, Diyi |
| author_facet | Manakul, Potsawee Gan, Woody Haosheng Ryan, Michael J. Khan, Ali Sartaz Sirichotedumrong, Warit Pipatanakul, Kunat Held, William Yang, Diyi |
| contents | Current speech evaluation suffers from two critical limitations: the need and difficulty of designing specialized systems targeting individual audio characteristics, and poor correlation between automatic evaluation methods and human preferences. This work presents a systematic study of Large Audio Model (LAM) as a Judge, AudioJudge, investigating whether it can provide a unified evaluation framework that addresses both challenges. We systematically explore AudioJudge across audio characteristic detection tasks, including pronunciation, speaking rate, speaker identification and speech quality, and system-level human preference simulation for automated benchmarking. We investigate different prompt engineering strategies, finding that audio concatenation combined with in-context learning significantly improves performance across both audio characteristic detection and human preference simulation tasks. We further introduce a multi-aspect ensemble AudioJudge to enable general-purpose multi-aspect audio evaluation. This method decomposes speech assessment into specialized judges for lexical content, speech quality, and paralinguistic features, achieving up to 0.91 Spearman correlation with human preferences on our system ranking benchmark. Robustness analysis reveals that while LAMs maintain strong performance under acoustic noise, they exhibit significant verbosity and positional biases that require careful mitigation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_12705 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluation Manakul, Potsawee Gan, Woody Haosheng Ryan, Michael J. Khan, Ali Sartaz Sirichotedumrong, Warit Pipatanakul, Kunat Held, William Yang, Diyi Computation and Language Sound Audio and Speech Processing Current speech evaluation suffers from two critical limitations: the need and difficulty of designing specialized systems targeting individual audio characteristics, and poor correlation between automatic evaluation methods and human preferences. This work presents a systematic study of Large Audio Model (LAM) as a Judge, AudioJudge, investigating whether it can provide a unified evaluation framework that addresses both challenges. We systematically explore AudioJudge across audio characteristic detection tasks, including pronunciation, speaking rate, speaker identification and speech quality, and system-level human preference simulation for automated benchmarking. We investigate different prompt engineering strategies, finding that audio concatenation combined with in-context learning significantly improves performance across both audio characteristic detection and human preference simulation tasks. We further introduce a multi-aspect ensemble AudioJudge to enable general-purpose multi-aspect audio evaluation. This method decomposes speech assessment into specialized judges for lexical content, speech quality, and paralinguistic features, achieving up to 0.91 Spearman correlation with human preferences on our system ranking benchmark. Robustness analysis reveals that while LAMs maintain strong performance under acoustic noise, they exhibit significant verbosity and positional biases that require careful mitigation. |
| title | AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluation |
| topic | Computation and Language Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2507.12705 |