SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processing
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914255443329024 |
|---|---|
| author | Ma, Ziyang Yang, Guanrou Chen, Wenxi Gao, Zhifu Du, Yexing Li, Xiquan Zheng, Zhisheng Zhu, Haina Zhuo, Jianheng Song, Zheshu Xu, Ruiyang Wang, Tiranrui Yang, Yifan Zhu, Yanqiao Niu, Zhikang Xue, Liumeng Ma, Yinghao Yuan, Ruibin Zhang, Shiliang Yu, Kai Chng, Eng Siong Chen, Xie |
| author_facet | Ma, Ziyang Yang, Guanrou Chen, Wenxi Gao, Zhifu Du, Yexing Li, Xiquan Zheng, Zhisheng Zhu, Haina Zhuo, Jianheng Song, Zheshu Xu, Ruiyang Wang, Tiranrui Yang, Yifan Zhu, Yanqiao Niu, Zhikang Xue, Liumeng Ma, Yinghao Yuan, Ruibin Zhang, Shiliang Yu, Kai Chng, Eng Siong Chen, Xie |
| contents | The recent surge in open-source Multimodal Large Language Models (MLLM) frameworks, such as LLaVA, provides a convenient kickoff for artificial intelligence developers and researchers. However, most of the MLLM frameworks take vision as the main input modality, and provide limited in-depth support for the modality of speech, audio, and music. This situation hinders the development of audio-language models, and forces researchers to spend a lot of effort on code writing and hyperparameter tuning. We present SLAM-LLM, an open-source deep learning framework designed to train customized MLLMs, focused on speech, language, audio, and music processing. SLAM-LLM provides a modular configuration of different encoders, projectors, LLMs, and parameter-efficient fine-tuning plugins. SLAM-LLM also includes detailed training and inference recipes for mainstream tasks, along with high-performance checkpoints like LLM-based Automatic Speech Recognition (ASR), Automated Audio Captioning (AAC), and Music Captioning (MC). Some of these recipes have already reached or are nearing state-of-the-art performance, and some relevant techniques have also been accepted by academic papers. We hope SLAM-LLM will accelerate iteration, development, data engineering, and model training for researchers. We are committed to continually pushing forward audio-based MLLMs through this open-source framework, and call on the community to contribute to the LLM-based speech, audio and music processing. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_09385 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processing Ma, Ziyang Yang, Guanrou Chen, Wenxi Gao, Zhifu Du, Yexing Li, Xiquan Zheng, Zhisheng Zhu, Haina Zhuo, Jianheng Song, Zheshu Xu, Ruiyang Wang, Tiranrui Yang, Yifan Zhu, Yanqiao Niu, Zhikang Xue, Liumeng Ma, Yinghao Yuan, Ruibin Zhang, Shiliang Yu, Kai Chng, Eng Siong Chen, Xie Sound Computation and Language Multimedia The recent surge in open-source Multimodal Large Language Models (MLLM) frameworks, such as LLaVA, provides a convenient kickoff for artificial intelligence developers and researchers. However, most of the MLLM frameworks take vision as the main input modality, and provide limited in-depth support for the modality of speech, audio, and music. This situation hinders the development of audio-language models, and forces researchers to spend a lot of effort on code writing and hyperparameter tuning. We present SLAM-LLM, an open-source deep learning framework designed to train customized MLLMs, focused on speech, language, audio, and music processing. SLAM-LLM provides a modular configuration of different encoders, projectors, LLMs, and parameter-efficient fine-tuning plugins. SLAM-LLM also includes detailed training and inference recipes for mainstream tasks, along with high-performance checkpoints like LLM-based Automatic Speech Recognition (ASR), Automated Audio Captioning (AAC), and Music Captioning (MC). Some of these recipes have already reached or are nearing state-of-the-art performance, and some relevant techniques have also been accepted by academic papers. We hope SLAM-LLM will accelerate iteration, development, data engineering, and model training for researchers. We are committed to continually pushing forward audio-based MLLMs through this open-source framework, and call on the community to contribute to the LLM-based speech, audio and music processing. |
| title | SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processing |
| topic | Sound Computation and Language Multimedia |
| url | https://arxiv.org/abs/2601.09385 |