Scaling Properties of Continuous Diffusion Spoken Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914510528315392 |
|---|---|
| author | Ramapuram, Jason Dhekane, Eeshan Gunesh Shidani, Amitis Busbridge, Dan Mazoure, Bogdan Gu, Zijin Webb, Russ Likhomanenko, Tatiana Jaitly, Navdeep |
| author_facet | Ramapuram, Jason Dhekane, Eeshan Gunesh Shidani, Amitis Busbridge, Dan Mazoure, Bogdan Gu, Zijin Webb, Russ Likhomanenko, Tatiana Jaitly, Navdeep |
| contents | Speech-only spoken language models (SLMs) lag behind text and text-speech models in performance, with recent discrete autoregressive (AR) SLMs indicating significant computational and data demands to match text models. Since discretizing continuous speech for AR creates bottlenecks, we explore whether continuous diffusion (CD) SLM is more viable. To quantify the SLMs linguistic quality, we introduce the phoneme Jensen-Shannon divergence (pJSD) metric. Our analysis reveals CD SLMs, mirroring AR behavior, exhibit scaling laws for validation loss and pJSD, and show optimal token-to-parameter ratios decreasing as compute scales. However, for the latter, loss becomes insensitive to choice of data and model sizes, showing potential for fast inference. Scaling CD SLMs to 16B parameters with tens of millions of hours of conversational data enables generation of emotive, prosodic, multi-speaker, multilingual speech, though achieving long-form coherence remains a significant challenge. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_24416 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Scaling Properties of Continuous Diffusion Spoken Language Models Ramapuram, Jason Dhekane, Eeshan Gunesh Shidani, Amitis Busbridge, Dan Mazoure, Bogdan Gu, Zijin Webb, Russ Likhomanenko, Tatiana Jaitly, Navdeep Computation and Language Artificial Intelligence Machine Learning Speech-only spoken language models (SLMs) lag behind text and text-speech models in performance, with recent discrete autoregressive (AR) SLMs indicating significant computational and data demands to match text models. Since discretizing continuous speech for AR creates bottlenecks, we explore whether continuous diffusion (CD) SLM is more viable. To quantify the SLMs linguistic quality, we introduce the phoneme Jensen-Shannon divergence (pJSD) metric. Our analysis reveals CD SLMs, mirroring AR behavior, exhibit scaling laws for validation loss and pJSD, and show optimal token-to-parameter ratios decreasing as compute scales. However, for the latter, loss becomes insensitive to choice of data and model sizes, showing potential for fast inference. Scaling CD SLMs to 16B parameters with tens of millions of hours of conversational data enables generation of emotive, prosodic, multi-speaker, multilingual speech, though achieving long-form coherence remains a significant challenge. |
| title | Scaling Properties of Continuous Diffusion Spoken Language Models |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2604.24416 |