Efficient Speech Language Modeling via Energy Distance in Continuous Latent Space
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914110421073920 |
|---|---|
| author | Ma, Zhengrui Feng, Yang Shao, Chenze Meng, Fandong Zhou, Jie Zhang, Min |
| author_facet | Ma, Zhengrui Feng, Yang Shao, Chenze Meng, Fandong Zhou, Jie Zhang, Min |
| contents | We introduce SLED, an alternative approach to speech language modeling by encoding speech waveforms into sequences of continuous latent representations and modeling them autoregressively using an energy distance objective. The energy distance offers an analytical measure of the distributional gap by contrasting simulated and target samples, enabling efficient training to capture the underlying continuous autoregressive distribution. By bypassing reliance on residual vector quantization, SLED avoids discretization errors and eliminates the need for the complicated hierarchical architectures common in existing speech language models. It simplifies the overall modeling pipeline while preserving the richness of speech information and maintaining inference efficiency. Empirical results demonstrate that SLED achieves strong performance in both zero-shot and streaming speech synthesis, showing its potential for broader applications in general-purpose speech language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_13181 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Efficient Speech Language Modeling via Energy Distance in Continuous Latent Space Ma, Zhengrui Feng, Yang Shao, Chenze Meng, Fandong Zhou, Jie Zhang, Min Computation and Language Sound Audio and Speech Processing We introduce SLED, an alternative approach to speech language modeling by encoding speech waveforms into sequences of continuous latent representations and modeling them autoregressively using an energy distance objective. The energy distance offers an analytical measure of the distributional gap by contrasting simulated and target samples, enabling efficient training to capture the underlying continuous autoregressive distribution. By bypassing reliance on residual vector quantization, SLED avoids discretization errors and eliminates the need for the complicated hierarchical architectures common in existing speech language models. It simplifies the overall modeling pipeline while preserving the richness of speech information and maintaining inference efficiency. Empirical results demonstrate that SLED achieves strong performance in both zero-shot and streaming speech synthesis, showing its potential for broader applications in general-purpose speech language models. |
| title | Efficient Speech Language Modeling via Energy Distance in Continuous Latent Space |
| topic | Computation and Language Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2505.13181 |