MELD: Mel-Spectrogram-Based Speech Language Modeling with Discrete Latent Variables
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917543816462336 |
|---|---|
| author | Yeh, Sung-Lin Zhou, Wei Keren, Gil Le, Duc Meng, Zhong Tang, Hao Mahadeokar, Jay Kalinli, Ozlem Mourachko, Alexandre |
| author_facet | Yeh, Sung-Lin Zhou, Wei Keren, Gil Le, Duc Meng, Zhong Tang, Hao Mahadeokar, Jay Kalinli, Ozlem Mourachko, Alexandre |
| contents | Recent speech language models rely on encoders that are optimized separately from autoregressive models. Since these encoders are unaware of the downstream objectives, the extracted representations may not be optimal for downstream tasks. To address this limitation, we introduce a discrete latent variable model on mel spectrograms that jointly optimizes the encoder and the speech language model. Joint optimization not only brings improvements over codec-based and other mel-spectrogram-based baselines on zero-shot Text-to-Speech (TTS) and Speech-to-Text (STT) tasks, but also effectively alleviates common issues in autoregressive mel-spectrogram modeling, such as prolonged silence generation and word omissions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_29859 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MELD: Mel-Spectrogram-Based Speech Language Modeling with Discrete Latent Variables Yeh, Sung-Lin Zhou, Wei Keren, Gil Le, Duc Meng, Zhong Tang, Hao Mahadeokar, Jay Kalinli, Ozlem Mourachko, Alexandre Audio and Speech Processing Computation and Language Recent speech language models rely on encoders that are optimized separately from autoregressive models. Since these encoders are unaware of the downstream objectives, the extracted representations may not be optimal for downstream tasks. To address this limitation, we introduce a discrete latent variable model on mel spectrograms that jointly optimizes the encoder and the speech language model. Joint optimization not only brings improvements over codec-based and other mel-spectrogram-based baselines on zero-shot Text-to-Speech (TTS) and Speech-to-Text (STT) tasks, but also effectively alleviates common issues in autoregressive mel-spectrogram modeling, such as prolonged silence generation and word omissions. |
| title | MELD: Mel-Spectrogram-Based Speech Language Modeling with Discrete Latent Variables |
| topic | Audio and Speech Processing Computation and Language |
| url | https://arxiv.org/abs/2605.29859 |