MMAU-Pro: A Challenging and Comprehensive Benchmark for Holistic Evaluation of Audio General Intelligence
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908494806908928 |
|---|---|
| author | Kumar, Sonal Sedláček, Šimon Lokegaonkar, Vaibhavi López, Fernando Yu, Wenyi Anand, Nishit Ryu, Hyeonggon Chen, Lichang Plička, Maxim Hlaváček, Miroslav Ellingwood, William Fineas Udupa, Sathvik Hou, Siyuan Ferner, Allison Barahona, Sara Bolaños, Cecilia Rahi, Satish Herrera-Alarcón, Laura Dixit, Satvik Patil, Siddhi Deshmukh, Soham Koroshinadze, Lasha Liu, Yao Perera, Leibny Paola Garcia Zanou, Eleni Stafylakis, Themos Chung, Joon Son Harwath, David Zhang, Chao Manocha, Dinesh Lozano-Diez, Alicia Kesiraju, Santosh Ghosh, Sreyan Duraiswami, Ramani |
| author_facet | Kumar, Sonal Sedláček, Šimon Lokegaonkar, Vaibhavi López, Fernando Yu, Wenyi Anand, Nishit Ryu, Hyeonggon Chen, Lichang Plička, Maxim Hlaváček, Miroslav Ellingwood, William Fineas Udupa, Sathvik Hou, Siyuan Ferner, Allison Barahona, Sara Bolaños, Cecilia Rahi, Satish Herrera-Alarcón, Laura Dixit, Satvik Patil, Siddhi Deshmukh, Soham Koroshinadze, Lasha Liu, Yao Perera, Leibny Paola Garcia Zanou, Eleni Stafylakis, Themos Chung, Joon Son Harwath, David Zhang, Chao Manocha, Dinesh Lozano-Diez, Alicia Kesiraju, Santosh Ghosh, Sreyan Duraiswami, Ramani |
| contents | Audio comprehension-including speech, non-speech sounds, and music-is essential for achieving human-level intelligence. Consequently, AI agents must demonstrate holistic audio understanding to qualify as generally intelligent. However, evaluating auditory intelligence comprehensively remains challenging. To address this gap, we introduce MMAU-Pro, the most comprehensive and rigorously curated benchmark for assessing audio intelligence in AI systems. MMAU-Pro contains 5,305 instances, where each instance has one or more audios paired with human expert-generated question-answer pairs, spanning speech, sound, music, and their combinations. Unlike existing benchmarks, MMAU-Pro evaluates auditory intelligence across 49 unique skills and multiple complex dimensions, including long-form audio comprehension, spatial audio reasoning, multi-audio understanding, among others. All questions are meticulously designed to require deliberate multi-hop reasoning, including both multiple-choice and open-ended response formats. Importantly, audio data is sourced directly ``from the wild" rather than from existing datasets with known distributions. We evaluate 22 leading open-source and proprietary multimodal AI models, revealing significant limitations: even state-of-the-art models such as Gemini 2.5 Flash and Audio Flamingo 3 achieve only 59.2% and 51.7% accuracy, respectively, approaching random performance in multiple categories. Our extensive analysis highlights specific shortcomings and provides novel insights, offering actionable perspectives for the community to enhance future AI systems' progression toward audio general intelligence. The benchmark and code is available at https://sonalkum.github.io/mmau-pro. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_13992 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MMAU-Pro: A Challenging and Comprehensive Benchmark for Holistic Evaluation of Audio General Intelligence Kumar, Sonal Sedláček, Šimon Lokegaonkar, Vaibhavi López, Fernando Yu, Wenyi Anand, Nishit Ryu, Hyeonggon Chen, Lichang Plička, Maxim Hlaváček, Miroslav Ellingwood, William Fineas Udupa, Sathvik Hou, Siyuan Ferner, Allison Barahona, Sara Bolaños, Cecilia Rahi, Satish Herrera-Alarcón, Laura Dixit, Satvik Patil, Siddhi Deshmukh, Soham Koroshinadze, Lasha Liu, Yao Perera, Leibny Paola Garcia Zanou, Eleni Stafylakis, Themos Chung, Joon Son Harwath, David Zhang, Chao Manocha, Dinesh Lozano-Diez, Alicia Kesiraju, Santosh Ghosh, Sreyan Duraiswami, Ramani Audio and Speech Processing Sound Audio comprehension-including speech, non-speech sounds, and music-is essential for achieving human-level intelligence. Consequently, AI agents must demonstrate holistic audio understanding to qualify as generally intelligent. However, evaluating auditory intelligence comprehensively remains challenging. To address this gap, we introduce MMAU-Pro, the most comprehensive and rigorously curated benchmark for assessing audio intelligence in AI systems. MMAU-Pro contains 5,305 instances, where each instance has one or more audios paired with human expert-generated question-answer pairs, spanning speech, sound, music, and their combinations. Unlike existing benchmarks, MMAU-Pro evaluates auditory intelligence across 49 unique skills and multiple complex dimensions, including long-form audio comprehension, spatial audio reasoning, multi-audio understanding, among others. All questions are meticulously designed to require deliberate multi-hop reasoning, including both multiple-choice and open-ended response formats. Importantly, audio data is sourced directly ``from the wild" rather than from existing datasets with known distributions. We evaluate 22 leading open-source and proprietary multimodal AI models, revealing significant limitations: even state-of-the-art models such as Gemini 2.5 Flash and Audio Flamingo 3 achieve only 59.2% and 51.7% accuracy, respectively, approaching random performance in multiple categories. Our extensive analysis highlights specific shortcomings and provides novel insights, offering actionable perspectives for the community to enhance future AI systems' progression toward audio general intelligence. The benchmark and code is available at https://sonalkum.github.io/mmau-pro. |
| title | MMAU-Pro: A Challenging and Comprehensive Benchmark for Holistic Evaluation of Audio General Intelligence |
| topic | Audio and Speech Processing Sound |
| url | https://arxiv.org/abs/2508.13992 |