AudioCapBench: Quick Evaluation on Audio Captioning across Sound, Music, and Speech
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908855392272384 |
|---|---|
| author | Qiu, Jielin Zhang, Jianguo Chen, Zixiang Yang, Liangwei Zhu, Ming Tan, Juntao Chen, Haolin Zhao, Wenting Murthy, Rithesh Ram, Roshan Prabhakar, Akshara Heinecke, Shelby Caiming Xiong Savarese, Silvio Wang, Huan |
| author_facet | Qiu, Jielin Zhang, Jianguo Chen, Zixiang Yang, Liangwei Zhu, Ming Tan, Juntao Chen, Haolin Zhao, Wenting Murthy, Rithesh Ram, Roshan Prabhakar, Akshara Heinecke, Shelby Caiming Xiong Savarese, Silvio Wang, Huan |
| contents | We introduce AudioCapBench, a benchmark for evaluating audio captioning capabilities of large multimodal models. \method covers three distinct audio domains, including environmental sound, music, and speech, with 1,000 curated evaluation samples drawn from established datasets. We evaluate 13 models across two providers (OpenAI, Google Gemini) using both reference-based metrics (METEOR, BLEU, ROUGE-L) and an LLM-as-Judge framework that scores predictions on three orthogonal dimensions: \textit{accuracy} (semantic correctness), \textit{completeness} (coverage of reference content), and \textit{hallucination} (absence of fabricated content). Our results reveal that Gemini models generally outperform OpenAI models on overall captioning quality, with Gemini~3~Pro achieving the highest overall score (6.00/10), while OpenAI models exhibit lower hallucination rates. All models perform best on speech captioning and worst on music captioning. We release the benchmark as well as evaluation code to facilitate reproducible audio understanding research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_23649 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | AudioCapBench: Quick Evaluation on Audio Captioning across Sound, Music, and Speech Qiu, Jielin Zhang, Jianguo Chen, Zixiang Yang, Liangwei Zhu, Ming Tan, Juntao Chen, Haolin Zhao, Wenting Murthy, Rithesh Ram, Roshan Prabhakar, Akshara Heinecke, Shelby Caiming Xiong Savarese, Silvio Wang, Huan Sound Artificial Intelligence We introduce AudioCapBench, a benchmark for evaluating audio captioning capabilities of large multimodal models. \method covers three distinct audio domains, including environmental sound, music, and speech, with 1,000 curated evaluation samples drawn from established datasets. We evaluate 13 models across two providers (OpenAI, Google Gemini) using both reference-based metrics (METEOR, BLEU, ROUGE-L) and an LLM-as-Judge framework that scores predictions on three orthogonal dimensions: \textit{accuracy} (semantic correctness), \textit{completeness} (coverage of reference content), and \textit{hallucination} (absence of fabricated content). Our results reveal that Gemini models generally outperform OpenAI models on overall captioning quality, with Gemini~3~Pro achieving the highest overall score (6.00/10), while OpenAI models exhibit lower hallucination rates. All models perform best on speech captioning and worst on music captioning. We release the benchmark as well as evaluation code to facilitate reproducible audio understanding research. |
| title | AudioCapBench: Quick Evaluation on Audio Captioning across Sound, Music, and Speech |
| topic | Sound Artificial Intelligence |
| url | https://arxiv.org/abs/2602.23649 |