VocalNet: Speech LLM with Multi-Token Prediction for Faster and High-Quality Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913802142875648 |
|---|---|
| author | Wang, Yuhao Liu, Heyang Cheng, Ziyang Wu, Ronghua Gu, Qunshan Wang, Yanfeng Wang, Yu |
| author_facet | Wang, Yuhao Liu, Heyang Cheng, Ziyang Wu, Ronghua Gu, Qunshan Wang, Yanfeng Wang, Yu |
| contents | Speech large language models (LLMs) have emerged as a prominent research focus in speech processing. We introduce VocalNet-1B and VocalNet-8B, a series of high-performance, low-latency speech LLMs enabled by a scalable and model-agnostic training framework designed for real-time voice interaction. Central to our contribution is the first application of multi-token prediction (MTP) to speech LLMs. This approach represents a paradigm shift from standard next-token prediction (NTP), offering simultaneous improvements in generation speed and quality. Informed by analysis of MTP's effect on speech generation and experimental comparisons, we designed a straightforward and highly effective MTP implementation. Experiments demonstrate that VocalNet performs on par with mainstream Omni LLMs even with limited training data, and significantly surpasses existing open-source speech LLMs. To foster reproducibility and community advancement, all model weights, inference code, training data, and framework implementations have been made publicly available at https://github.com/SJTU-OmniAgent/VocalNet |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_04060 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | VocalNet: Speech LLM with Multi-Token Prediction for Faster and High-Quality Generation Wang, Yuhao Liu, Heyang Cheng, Ziyang Wu, Ronghua Gu, Qunshan Wang, Yanfeng Wang, Yu Computation and Language Artificial Intelligence Sound Audio and Speech Processing Speech large language models (LLMs) have emerged as a prominent research focus in speech processing. We introduce VocalNet-1B and VocalNet-8B, a series of high-performance, low-latency speech LLMs enabled by a scalable and model-agnostic training framework designed for real-time voice interaction. Central to our contribution is the first application of multi-token prediction (MTP) to speech LLMs. This approach represents a paradigm shift from standard next-token prediction (NTP), offering simultaneous improvements in generation speed and quality. Informed by analysis of MTP's effect on speech generation and experimental comparisons, we designed a straightforward and highly effective MTP implementation. Experiments demonstrate that VocalNet performs on par with mainstream Omni LLMs even with limited training data, and significantly surpasses existing open-source speech LLMs. To foster reproducibility and community advancement, all model weights, inference code, training data, and framework implementations have been made publicly available at https://github.com/SJTU-OmniAgent/VocalNet |
| title | VocalNet: Speech LLM with Multi-Token Prediction for Faster and High-Quality Generation |
| topic | Computation and Language Artificial Intelligence Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2504.04060 |