DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908901082923008 |
|---|---|
| author | Lu, Ke-Han Chen, Zhehuai Fu, Szu-Wei Yang, Chao-Han Huck Huang, Sung-Feng Yang, Chih-Kai Yu, Chee-En Chen, Chun-Wei Chen, Wei-Chih Huang, Chien-yu Lin, Yi-Cheng Lin, Yu-Xiang Fu, Chi-An Kuan, Chun-Yi Ren, Wenze Chen, Xuanjun Huang, Wei-Ping Hu, En-Pei Lin, Tzu-Quan Wu, Yuan-Kuei Huang, Kuan-Po Huang, Hsiao-Ying Chou, Huang-Cheng Chang, Kai-Wei Chiang, Cheng-Han Ginsburg, Boris Wang, Yu-Chiang Frank Lee, Hung-yi |
| author_facet | Lu, Ke-Han Chen, Zhehuai Fu, Szu-Wei Yang, Chao-Han Huck Huang, Sung-Feng Yang, Chih-Kai Yu, Chee-En Chen, Chun-Wei Chen, Wei-Chih Huang, Chien-yu Lin, Yi-Cheng Lin, Yu-Xiang Fu, Chi-An Kuan, Chun-Yi Ren, Wenze Chen, Xuanjun Huang, Wei-Ping Hu, En-Pei Lin, Tzu-Quan Wu, Yuan-Kuei Huang, Kuan-Po Huang, Hsiao-Ying Chou, Huang-Cheng Chang, Kai-Wei Chiang, Cheng-Han Ginsburg, Boris Wang, Yu-Chiang Frank Lee, Hung-yi |
| contents | We introduce DeSTA2.5-Audio, a general-purpose Large Audio Language Model (LALM) designed for robust auditory perception and instruction-following. Recent LALMs augment Large Language Models (LLMs) with auditory capabilities by training on large-scale audio-instruction datasets. However, existing LALMs have often suffered from the catastrophic forgetting of the LLM's original abilities. Therefore, balancing knowledge retention and audio perception has become a critical challenge. To address this, we revisit the data construction pipeline and propose a self-generated cross-modal alignment strategy in which the backbone LLM generates its own training targets, named DeSTA. This approach aims at preserving the LLM's native language proficiency thereby enabling zero-shot generalization without task-specific tuning. We construct DeSTA-AQA5M, a large-scale, task-agnostic dataset containing 5 million training samples derived from 7,000 hours of audio spanning 50 diverse datasets, including speech, environmental sounds, and music. DeSTA2.5-Audio achieves state-of-the-art or competitive performance across a wide range of audio-language benchmarks, including Dynamic-SUPERB, MMAU, SAKURA, Speech-IFEval, and VoiceBench. Comprehensive comparative studies demonstrate that our self-generated strategy outperforms existing training strategies. Our findings underscore the importance of carefully designed data construction in LALM development and offer practical insights for building robust, general-purpose LALMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_02768 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment Lu, Ke-Han Chen, Zhehuai Fu, Szu-Wei Yang, Chao-Han Huck Huang, Sung-Feng Yang, Chih-Kai Yu, Chee-En Chen, Chun-Wei Chen, Wei-Chih Huang, Chien-yu Lin, Yi-Cheng Lin, Yu-Xiang Fu, Chi-An Kuan, Chun-Yi Ren, Wenze Chen, Xuanjun Huang, Wei-Ping Hu, En-Pei Lin, Tzu-Quan Wu, Yuan-Kuei Huang, Kuan-Po Huang, Hsiao-Ying Chou, Huang-Cheng Chang, Kai-Wei Chiang, Cheng-Han Ginsburg, Boris Wang, Yu-Chiang Frank Lee, Hung-yi Audio and Speech Processing Computation and Language Sound We introduce DeSTA2.5-Audio, a general-purpose Large Audio Language Model (LALM) designed for robust auditory perception and instruction-following. Recent LALMs augment Large Language Models (LLMs) with auditory capabilities by training on large-scale audio-instruction datasets. However, existing LALMs have often suffered from the catastrophic forgetting of the LLM's original abilities. Therefore, balancing knowledge retention and audio perception has become a critical challenge. To address this, we revisit the data construction pipeline and propose a self-generated cross-modal alignment strategy in which the backbone LLM generates its own training targets, named DeSTA. This approach aims at preserving the LLM's native language proficiency thereby enabling zero-shot generalization without task-specific tuning. We construct DeSTA-AQA5M, a large-scale, task-agnostic dataset containing 5 million training samples derived from 7,000 hours of audio spanning 50 diverse datasets, including speech, environmental sounds, and music. DeSTA2.5-Audio achieves state-of-the-art or competitive performance across a wide range of audio-language benchmarks, including Dynamic-SUPERB, MMAU, SAKURA, Speech-IFEval, and VoiceBench. Comprehensive comparative studies demonstrate that our self-generated strategy outperforms existing training strategies. Our findings underscore the importance of carefully designed data construction in LALM development and offer practical insights for building robust, general-purpose LALMs. |
| title | DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment |
| topic | Audio and Speech Processing Computation and Language Sound |
| url | https://arxiv.org/abs/2507.02768 |