Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909688290869248 |
|---|---|
| author | Yang, Shu-wen Kim, Byeonggeun Huang, Kuan-Po Tang, Qingming Phan, Huy Lu, Bo-Ru Sundar, Harsha Ghosh, Shalini Lee, Hung-yi Kao, Chieh-Chi Wang, Chao |
| author_facet | Yang, Shu-wen Kim, Byeonggeun Huang, Kuan-Po Tang, Qingming Phan, Huy Lu, Bo-Ru Sundar, Harsha Ghosh, Shalini Lee, Hung-yi Kao, Chieh-Chi Wang, Chao |
| contents | Autoregressive next-token prediction with the Transformer decoder has become a de facto standard in large language models (LLMs), achieving remarkable success in Natural Language Processing (NLP) at scale. Extending this paradigm to audio poses unique challenges due to its inherently continuous nature. We research audio generation with a causal language model (LM) without discrete tokens. We leverage token-wise diffusion to model the continuous distribution of the next continuous-valued token. Our approach delivers significant improvements over previous discrete solution, AudioGen, achieving 20% and 40% relative gains on AudioCaps in Frechet Audio Distance (FAD) and Kullback-Leibler (KL) divergence, respectively. Additionally, we propose a novel masked next-token prediction task that incorporates masked prediction into the causal LM framework. On AudioCaps, the innovation yields 41% and 33% relative FAD improvements over AudioGen Base (285M) and AudioGen Large (1B) models, respectively, and is on par with the state-of-the-art (SOTA) diffusion models. Furthermore, we achieve these results with significantly fewer parameters -- 193M for our Base and 462M for our Large models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_09834 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction Yang, Shu-wen Kim, Byeonggeun Huang, Kuan-Po Tang, Qingming Phan, Huy Lu, Bo-Ru Sundar, Harsha Ghosh, Shalini Lee, Hung-yi Kao, Chieh-Chi Wang, Chao Audio and Speech Processing Computer Vision and Pattern Recognition Sound Autoregressive next-token prediction with the Transformer decoder has become a de facto standard in large language models (LLMs), achieving remarkable success in Natural Language Processing (NLP) at scale. Extending this paradigm to audio poses unique challenges due to its inherently continuous nature. We research audio generation with a causal language model (LM) without discrete tokens. We leverage token-wise diffusion to model the continuous distribution of the next continuous-valued token. Our approach delivers significant improvements over previous discrete solution, AudioGen, achieving 20% and 40% relative gains on AudioCaps in Frechet Audio Distance (FAD) and Kullback-Leibler (KL) divergence, respectively. Additionally, we propose a novel masked next-token prediction task that incorporates masked prediction into the causal LM framework. On AudioCaps, the innovation yields 41% and 33% relative FAD improvements over AudioGen Base (285M) and AudioGen Large (1B) models, respectively, and is on par with the state-of-the-art (SOTA) diffusion models. Furthermore, we achieve these results with significantly fewer parameters -- 193M for our Base and 462M for our Large models. |
| title | Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction |
| topic | Audio and Speech Processing Computer Vision and Pattern Recognition Sound |
| url | https://arxiv.org/abs/2507.09834 |