Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2507.23511 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910208362545152 |
|---|---|
| author | Niu, Yadong Wang, Tianzi Dinkel, Heinrich Sun, Xingwei Zhou, Jiahao Li, Gang Liu, Jizhong Liu, Xunying Zhang, Junbo Luan, Jian |
| author_facet | Niu, Yadong Wang, Tianzi Dinkel, Heinrich Sun, Xingwei Zhou, Jiahao Li, Gang Liu, Jizhong Liu, Xunying Zhang, Junbo Luan, Jian |
| contents | While large audio-language models have advanced open-ended audio understanding, they still fall short of nuanced human-level comprehension. This gap persists largely because current benchmarks, limited by data annotations and evaluation metrics, fail to reliably distinguish between generic and highly detailed model outputs. To this end, this work introduces MECAT, a Multi-Expert Constructed Benchmark for Fine-Grained Audio Understanding Tasks. Generated via a pipeline that integrates analysis from specialized expert models with Chain-of-Thought large language model reasoning, MECAT provides multi-perspective, fine-grained captions and open-set question-answering pairs. The benchmark is complemented by a novel metric: DATE (Discriminative-Enhanced Audio Text Evaluation). This metric penalizes generic terms and rewards detailed descriptions by combining single-sample semantic similarity with cross-sample discriminability. A comprehensive evaluation of state-of-the-art audio models is also presented, providing new insights into their current capabilities and limitations. The data and code are available at https://github.com/xiaomi-research/mecat |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_23511 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Tasks Niu, Yadong Wang, Tianzi Dinkel, Heinrich Sun, Xingwei Zhou, Jiahao Li, Gang Liu, Jizhong Liu, Xunying Zhang, Junbo Luan, Jian Audio and Speech Processing Artificial Intelligence Computation and Language Sound While large audio-language models have advanced open-ended audio understanding, they still fall short of nuanced human-level comprehension. This gap persists largely because current benchmarks, limited by data annotations and evaluation metrics, fail to reliably distinguish between generic and highly detailed model outputs. To this end, this work introduces MECAT, a Multi-Expert Constructed Benchmark for Fine-Grained Audio Understanding Tasks. Generated via a pipeline that integrates analysis from specialized expert models with Chain-of-Thought large language model reasoning, MECAT provides multi-perspective, fine-grained captions and open-set question-answering pairs. The benchmark is complemented by a novel metric: DATE (Discriminative-Enhanced Audio Text Evaluation). This metric penalizes generic terms and rewards detailed descriptions by combining single-sample semantic similarity with cross-sample discriminability. A comprehensive evaluation of state-of-the-art audio models is also presented, providing new insights into their current capabilities and limitations. The data and code are available at https://github.com/xiaomi-research/mecat |
| title | MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Tasks |
| topic | Audio and Speech Processing Artificial Intelligence Computation and Language Sound |
| url | https://arxiv.org/abs/2507.23511 |