Measuring Audio's Impact on Correctness: Audio-Contribution-Aware Post-Training of Large Audio Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917323849334784 |
|---|---|
| author | He, Haolin Du, Xingjian Sun, Renhe Dai, Zheqi Xiao, Yujia Yang, Mingru Zhou, Jiayi Li, Xiquan Liu, Zhengxi Liang, Zining Wu, Chunyat He, Qianhua Lee, Tan Chen, Xie Zheng, Wei-Long Wang, Weiqiang Plumbley, Mark Liu, Jian Kong, Qiuqiang |
| author_facet | He, Haolin Du, Xingjian Sun, Renhe Dai, Zheqi Xiao, Yujia Yang, Mingru Zhou, Jiayi Li, Xiquan Liu, Zhengxi Liang, Zining Wu, Chunyat He, Qianhua Lee, Tan Chen, Xie Zheng, Wei-Long Wang, Weiqiang Plumbley, Mark Liu, Jian Kong, Qiuqiang |
| contents | Large Audio Language Models (LALMs) represent an important frontier in multimodal AI, addressing diverse audio tasks. Recently, post-training of LALMs has received increasing attention due to significant performance improvements over foundation models. While single-stage post-training such as reinforcement learning (RL) has demonstrated promising results, multi-stage approaches such as supervised fine-tuning (SFT) followed by RL remain suboptimal. The allocation of data across multiple training stages to maximize LALM capabilities has not been fully explored, and large-scale, high-quality datasets for such research are also lacking. To address these problems, we firstly present AudioMCQ, a comprehensive audio multiple-choice question dataset comprising 571k samples with two kinds of chain-of-thought annotations. Secondly, we investigate the prevalent zero audio-contribution phenomenon in LALMs, where models derive correct answers solely from textual information without processing audio content. We propose Audio-Contribution Filtering to partition data into weak and strong audio-contribution subsets. Based on these insights, we develop two effective post-training paradigms: Weak-to-Strong (SFT on weak audio-contribution data followed by RL on strong audio-contribution data) and Mixed-to-Strong (SFT on mixed audio-contribution data followed by RL on strong audio-contribution data). We achieve first place in the DCASE 2025 Audio-Question-Answering challenge by using AudioMCQ. Additionally, leveraging our dataset with different training strategies, we achieve 78.2\% on MMAU-test-mini, 75.6\% on MMAU, 67.1\% on MMAR, and 70.7\% on MMSU, establishing new state-of-the-art performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_21060 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Measuring Audio's Impact on Correctness: Audio-Contribution-Aware Post-Training of Large Audio Language Models He, Haolin Du, Xingjian Sun, Renhe Dai, Zheqi Xiao, Yujia Yang, Mingru Zhou, Jiayi Li, Xiquan Liu, Zhengxi Liang, Zining Wu, Chunyat He, Qianhua Lee, Tan Chen, Xie Zheng, Wei-Long Wang, Weiqiang Plumbley, Mark Liu, Jian Kong, Qiuqiang Audio and Speech Processing Large Audio Language Models (LALMs) represent an important frontier in multimodal AI, addressing diverse audio tasks. Recently, post-training of LALMs has received increasing attention due to significant performance improvements over foundation models. While single-stage post-training such as reinforcement learning (RL) has demonstrated promising results, multi-stage approaches such as supervised fine-tuning (SFT) followed by RL remain suboptimal. The allocation of data across multiple training stages to maximize LALM capabilities has not been fully explored, and large-scale, high-quality datasets for such research are also lacking. To address these problems, we firstly present AudioMCQ, a comprehensive audio multiple-choice question dataset comprising 571k samples with two kinds of chain-of-thought annotations. Secondly, we investigate the prevalent zero audio-contribution phenomenon in LALMs, where models derive correct answers solely from textual information without processing audio content. We propose Audio-Contribution Filtering to partition data into weak and strong audio-contribution subsets. Based on these insights, we develop two effective post-training paradigms: Weak-to-Strong (SFT on weak audio-contribution data followed by RL on strong audio-contribution data) and Mixed-to-Strong (SFT on mixed audio-contribution data followed by RL on strong audio-contribution data). We achieve first place in the DCASE 2025 Audio-Question-Answering challenge by using AudioMCQ. Additionally, leveraging our dataset with different training strategies, we achieve 78.2\% on MMAU-test-mini, 75.6\% on MMAU, 67.1\% on MMAR, and 70.7\% on MMSU, establishing new state-of-the-art performance. |
| title | Measuring Audio's Impact on Correctness: Audio-Contribution-Aware Post-Training of Large Audio Language Models |
| topic | Audio and Speech Processing |
| url | https://arxiv.org/abs/2509.21060 |