VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Model
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912662906994688 |
|---|---|
| author | Long, Zuwei Shen, Yunhang Fu, Chaoyou Gao, Heting Li, Lijiang Chen, Peixian Zhang, Mengdan Shao, Hang Li, Jian Peng, Jinlong Cao, Haoyu Li, Ke Ji, Rongrong Sun, Xing |
| author_facet | Long, Zuwei Shen, Yunhang Fu, Chaoyou Gao, Heting Li, Lijiang Chen, Peixian Zhang, Mengdan Shao, Hang Li, Jian Peng, Jinlong Cao, Haoyu Li, Ke Ji, Rongrong Sun, Xing |
| contents | With the growing requirement for natural human-computer interaction, speech-based systems receive increasing attention as speech is one of the most common forms of daily communication. However, the existing speech models still experience high latency when generating the first audio token during streaming, which poses a significant bottleneck for deployment. To address this issue, we propose VITA-Audio, an end-to-end large speech model with fast audio-text token generation. Specifically, we introduce a lightweight Multiple Cross-modal Token Prediction (MCTP) module that efficiently generates multiple audio tokens within a single model forward pass, which not only accelerates the inference but also significantly reduces the latency for generating the first audio in streaming scenarios. In addition, a four-stage progressive training strategy is explored to achieve model acceleration with minimal loss of speech quality. To our knowledge, VITA-Audio is the first multi-modal large language model capable of generating audio output during the first forward pass, enabling real-time conversational capabilities with minimal latency. VITA-Audio is fully reproducible and is trained on open-source data only. Experimental results demonstrate that our model achieves an inference speedup of 3~5x at the 7B parameter scale, but also significantly outperforms open-source models of similar model size on multiple benchmarks for automatic speech recognition (ASR), text-to-speech (TTS), and spoken question answering (SQA) tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_03739 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Model Long, Zuwei Shen, Yunhang Fu, Chaoyou Gao, Heting Li, Lijiang Chen, Peixian Zhang, Mengdan Shao, Hang Li, Jian Peng, Jinlong Cao, Haoyu Li, Ke Ji, Rongrong Sun, Xing Computation and Language Artificial Intelligence With the growing requirement for natural human-computer interaction, speech-based systems receive increasing attention as speech is one of the most common forms of daily communication. However, the existing speech models still experience high latency when generating the first audio token during streaming, which poses a significant bottleneck for deployment. To address this issue, we propose VITA-Audio, an end-to-end large speech model with fast audio-text token generation. Specifically, we introduce a lightweight Multiple Cross-modal Token Prediction (MCTP) module that efficiently generates multiple audio tokens within a single model forward pass, which not only accelerates the inference but also significantly reduces the latency for generating the first audio in streaming scenarios. In addition, a four-stage progressive training strategy is explored to achieve model acceleration with minimal loss of speech quality. To our knowledge, VITA-Audio is the first multi-modal large language model capable of generating audio output during the first forward pass, enabling real-time conversational capabilities with minimal latency. VITA-Audio is fully reproducible and is trained on open-source data only. Experimental results demonstrate that our model achieves an inference speedup of 3~5x at the 7B parameter scale, but also significantly outperforms open-source models of similar model size on multiple benchmarks for automatic speech recognition (ASR), text-to-speech (TTS), and spoken question answering (SQA) tasks. |
| title | VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Model |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2505.03739 |