MoE Adapter for Large Audio Language Models: Sparsity, Disentanglement, and Gradient-Conflict-Free
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914239146360832 |
|---|---|
| author | Lei, Yishu He, Shuwei Hu, Jing Zhang, Dan Luo, Xianlong Zhu, Danxiang Feng, Shikun Liu, Rui He, Jingzhou Sun, Yu Wu, Hua Wang, Haifeng |
| author_facet | Lei, Yishu He, Shuwei Hu, Jing Zhang, Dan Luo, Xianlong Zhu, Danxiang Feng, Shikun Liu, Rui He, Jingzhou Sun, Yu Wu, Hua Wang, Haifeng |
| contents | Extending the input modality of Large Language Models~(LLMs) to the audio domain is essential for achieving comprehensive multimodal perception. However, it is well-known that acoustic information is intrinsically \textit{heterogeneous}, entangling attributes such as speech, music, and environmental context. Existing research is limited to a dense, parameter-shared adapter to model these diverse patterns, which induces \textit{gradient conflict} during optimization, as parameter updates required for distinct attributes contradict each other. To address this limitation, we introduce the \textit{\textbf{MoE-Adapter}}, a sparse Mixture-of-Experts~(MoE) architecture designed to decouple acoustic information. Specifically, it employs a dynamic gating mechanism that routes audio tokens to specialized experts capturing complementary feature subspaces while retaining shared experts for global context, thereby mitigating gradient conflicts and enabling fine-grained feature learning. Comprehensive experiments show that the MoE-Adapter achieves superior performance on both audio semantic and paralinguistic tasks, consistently outperforming dense linear baselines with comparable computational costs. Furthermore, we will release the related code and models to facilitate future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_02967 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MoE Adapter for Large Audio Language Models: Sparsity, Disentanglement, and Gradient-Conflict-Free Lei, Yishu He, Shuwei Hu, Jing Zhang, Dan Luo, Xianlong Zhu, Danxiang Feng, Shikun Liu, Rui He, Jingzhou Sun, Yu Wu, Hua Wang, Haifeng Sound Artificial Intelligence Audio and Speech Processing Extending the input modality of Large Language Models~(LLMs) to the audio domain is essential for achieving comprehensive multimodal perception. However, it is well-known that acoustic information is intrinsically \textit{heterogeneous}, entangling attributes such as speech, music, and environmental context. Existing research is limited to a dense, parameter-shared adapter to model these diverse patterns, which induces \textit{gradient conflict} during optimization, as parameter updates required for distinct attributes contradict each other. To address this limitation, we introduce the \textit{\textbf{MoE-Adapter}}, a sparse Mixture-of-Experts~(MoE) architecture designed to decouple acoustic information. Specifically, it employs a dynamic gating mechanism that routes audio tokens to specialized experts capturing complementary feature subspaces while retaining shared experts for global context, thereby mitigating gradient conflicts and enabling fine-grained feature learning. Comprehensive experiments show that the MoE-Adapter achieves superior performance on both audio semantic and paralinguistic tasks, consistently outperforming dense linear baselines with comparable computational costs. Furthermore, we will release the related code and models to facilitate future research. |
| title | MoE Adapter for Large Audio Language Models: Sparsity, Disentanglement, and Gradient-Conflict-Free |
| topic | Sound Artificial Intelligence Audio and Speech Processing |
| url | https://arxiv.org/abs/2601.02967 |