ACAVCaps: Enabling large-scale training for fine-grained and diverse audio understanding
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918407695237120 |
|---|---|
| author | Niu, Yadong Wang, Tianzi Dinkel, Heinrich Sun, Xingwei Zhou, Jiahao Li, Gang Liu, Jizhong Zhang, Junbo Luan, Jian |
| author_facet | Niu, Yadong Wang, Tianzi Dinkel, Heinrich Sun, Xingwei Zhou, Jiahao Li, Gang Liu, Jizhong Zhang, Junbo Luan, Jian |
| contents | General audio understanding is a fundamental goal for large audio-language models, with audio captioning serving as a cornerstone task for their development. However, progress in this domain is hindered by existing datasets, which lack the scale and descriptive granularity required to train truly versatile models. To address this gap, we introduce ACAVCaps, a new large-scale, fine-grained, and multi-faceted audio captioning dataset. Derived from the ACAV100M collection, ACAVCaps is constructed using a multi-expert pipeline that analyzes audio from diverse perspectives-including speech, music, and acoustic properties-which are then synthesized into rich, detailed descriptions by a large language model. Experimental results demonstrate that models pre-trained on ACAVCaps exhibit substantially stronger generalization capabilities on various downstream tasks compared to those trained on other leading captioning datasets. The dataset is available at https://github.com/xiaomi-research/acavcaps. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_24038 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | ACAVCaps: Enabling large-scale training for fine-grained and diverse audio understanding Niu, Yadong Wang, Tianzi Dinkel, Heinrich Sun, Xingwei Zhou, Jiahao Li, Gang Liu, Jizhong Zhang, Junbo Luan, Jian Audio and Speech Processing Sound General audio understanding is a fundamental goal for large audio-language models, with audio captioning serving as a cornerstone task for their development. However, progress in this domain is hindered by existing datasets, which lack the scale and descriptive granularity required to train truly versatile models. To address this gap, we introduce ACAVCaps, a new large-scale, fine-grained, and multi-faceted audio captioning dataset. Derived from the ACAV100M collection, ACAVCaps is constructed using a multi-expert pipeline that analyzes audio from diverse perspectives-including speech, music, and acoustic properties-which are then synthesized into rich, detailed descriptions by a large language model. Experimental results demonstrate that models pre-trained on ACAVCaps exhibit substantially stronger generalization capabilities on various downstream tasks compared to those trained on other leading captioning datasets. The dataset is available at https://github.com/xiaomi-research/acavcaps. |
| title | ACAVCaps: Enabling large-scale training for fine-grained and diverse audio understanding |
| topic | Audio and Speech Processing Sound |
| url | https://arxiv.org/abs/2603.24038 |