MinMo: A Multimodal Large Language Model for Seamless Voice Interaction
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912184172281856 |
|---|---|
| author | Chen, Qian Chen, Yafeng Chen, Yanni Chen, Mengzhe Chen, Yingda Deng, Chong Du, Zhihao Gao, Ruize Gao, Changfeng Gao, Zhifu Li, Yabin Lv, Xiang Liu, Jiaqing Luo, Haoneng Ma, Bin Ni, Chongjia Shi, Xian Tang, Jialong Wang, Hui Wang, Hao Wang, Wen Wang, Yuxuan Xu, Yunlan Yu, Fan Yan, Zhijie Yang, Yexin Yang, Baosong Yang, Xian Yang, Guanrou Zhao, Tianyu Zhang, Qinglin Zhang, Shiliang Zhao, Nan Zhang, Pei Zhang, Chong Zhou, Jinren |
| author_facet | Chen, Qian Chen, Yafeng Chen, Yanni Chen, Mengzhe Chen, Yingda Deng, Chong Du, Zhihao Gao, Ruize Gao, Changfeng Gao, Zhifu Li, Yabin Lv, Xiang Liu, Jiaqing Luo, Haoneng Ma, Bin Ni, Chongjia Shi, Xian Tang, Jialong Wang, Hui Wang, Hao Wang, Wen Wang, Yuxuan Xu, Yunlan Yu, Fan Yan, Zhijie Yang, Yexin Yang, Baosong Yang, Xian Yang, Guanrou Zhao, Tianyu Zhang, Qinglin Zhang, Shiliang Zhao, Nan Zhang, Pei Zhang, Chong Zhou, Jinren |
| contents | Recent advancements in large language models (LLMs) and multimodal speech-text models have laid the groundwork for seamless voice interactions, enabling real-time, natural, and human-like conversations. Previous models for voice interactions are categorized as native and aligned. Native models integrate speech and text processing in one framework but struggle with issues like differing sequence lengths and insufficient pre-training. Aligned models maintain text LLM capabilities but are often limited by small datasets and a narrow focus on speech tasks. In this work, we introduce MinMo, a Multimodal Large Language Model with approximately 8B parameters for seamless voice interaction. We address the main limitations of prior aligned multimodal models. We train MinMo through multiple stages of speech-to-text alignment, text-to-speech alignment, speech-to-speech alignment, and duplex interaction alignment, on 1.4 million hours of diverse speech data and a broad range of speech tasks. After the multi-stage training, MinMo achieves state-of-the-art performance across various benchmarks for voice comprehension and generation while maintaining the capabilities of text LLMs, and also facilitates full-duplex conversation, that is, simultaneous two-way communication between the user and the system. Moreover, we propose a novel and simple voice decoder that outperforms prior models in voice generation. The enhanced instruction-following capabilities of MinMo supports controlling speech generation based on user instructions, with various nuances including emotions, dialects, and speaking rates, and mimicking specific voices. For MinMo, the speech-to-text latency is approximately 100ms, full-duplex latency is approximately 600ms in theory and 800ms in practice. The MinMo project web page is https://funaudiollm.github.io/minmo, and the code and models will be released soon. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_06282 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MinMo: A Multimodal Large Language Model for Seamless Voice Interaction Chen, Qian Chen, Yafeng Chen, Yanni Chen, Mengzhe Chen, Yingda Deng, Chong Du, Zhihao Gao, Ruize Gao, Changfeng Gao, Zhifu Li, Yabin Lv, Xiang Liu, Jiaqing Luo, Haoneng Ma, Bin Ni, Chongjia Shi, Xian Tang, Jialong Wang, Hui Wang, Hao Wang, Wen Wang, Yuxuan Xu, Yunlan Yu, Fan Yan, Zhijie Yang, Yexin Yang, Baosong Yang, Xian Yang, Guanrou Zhao, Tianyu Zhang, Qinglin Zhang, Shiliang Zhao, Nan Zhang, Pei Zhang, Chong Zhou, Jinren Computation and Language Artificial Intelligence Human-Computer Interaction Sound Audio and Speech Processing Recent advancements in large language models (LLMs) and multimodal speech-text models have laid the groundwork for seamless voice interactions, enabling real-time, natural, and human-like conversations. Previous models for voice interactions are categorized as native and aligned. Native models integrate speech and text processing in one framework but struggle with issues like differing sequence lengths and insufficient pre-training. Aligned models maintain text LLM capabilities but are often limited by small datasets and a narrow focus on speech tasks. In this work, we introduce MinMo, a Multimodal Large Language Model with approximately 8B parameters for seamless voice interaction. We address the main limitations of prior aligned multimodal models. We train MinMo through multiple stages of speech-to-text alignment, text-to-speech alignment, speech-to-speech alignment, and duplex interaction alignment, on 1.4 million hours of diverse speech data and a broad range of speech tasks. After the multi-stage training, MinMo achieves state-of-the-art performance across various benchmarks for voice comprehension and generation while maintaining the capabilities of text LLMs, and also facilitates full-duplex conversation, that is, simultaneous two-way communication between the user and the system. Moreover, we propose a novel and simple voice decoder that outperforms prior models in voice generation. The enhanced instruction-following capabilities of MinMo supports controlling speech generation based on user instructions, with various nuances including emotions, dialects, and speaking rates, and mimicking specific voices. For MinMo, the speech-to-text latency is approximately 100ms, full-duplex latency is approximately 600ms in theory and 800ms in practice. The MinMo project web page is https://funaudiollm.github.io/minmo, and the code and models will be released soon. |
| title | MinMo: A Multimodal Large Language Model for Seamless Voice Interaction |
| topic | Computation and Language Artificial Intelligence Human-Computer Interaction Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2501.06282 |