Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.22362 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909663123996672 |
|---|---|
| author | Yang, Yang Li, Yunpeng Sung, George Shih, Shao-Fu Dooley, Craig Centazzo, Alessio Rajeswaran, Ramanan |
| author_facet | Yang, Yang Li, Yunpeng Sung, George Shih, Shao-Fu Dooley, Craig Centazzo, Alessio Rajeswaran, Ramanan |
| contents | Token-based language modeling is a prominent approach for speech generation, where tokens are obtained by quantizing features from self-supervised learning (SSL) models and extracting codes from neural speech codecs, generally referred to as semantic tokens and acoustic tokens. These tokens are often modeled autoregressively, with the inference speed being constrained by the token rate. In this work, we propose DiffSoundStream, a solution that improves the efficiency of speech tokenization in non-streaming scenarios through two techniques: (1) conditioning the neural codec on semantic tokens to minimize redundancy between semantic and acoustic tokens, and (2) leveraging latent diffusion models to synthesize high-quality waveforms from semantic and coarse-level acoustic tokens. Experiments show that at 50 tokens per second, DiffSoundStream achieves speech quality on par with a standard SoundStream model operating at twice the token rate. Additionally, we achieve step-size distillation using just four diffusion sampling steps with only a minor quality loss. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_22362 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DiffSoundStream: Efficient Speech Tokenization via Diffusion Decoding Yang, Yang Li, Yunpeng Sung, George Shih, Shao-Fu Dooley, Craig Centazzo, Alessio Rajeswaran, Ramanan Audio and Speech Processing Machine Learning Token-based language modeling is a prominent approach for speech generation, where tokens are obtained by quantizing features from self-supervised learning (SSL) models and extracting codes from neural speech codecs, generally referred to as semantic tokens and acoustic tokens. These tokens are often modeled autoregressively, with the inference speed being constrained by the token rate. In this work, we propose DiffSoundStream, a solution that improves the efficiency of speech tokenization in non-streaming scenarios through two techniques: (1) conditioning the neural codec on semantic tokens to minimize redundancy between semantic and acoustic tokens, and (2) leveraging latent diffusion models to synthesize high-quality waveforms from semantic and coarse-level acoustic tokens. Experiments show that at 50 tokens per second, DiffSoundStream achieves speech quality on par with a standard SoundStream model operating at twice the token rate. Additionally, we achieve step-size distillation using just four diffusion sampling steps with only a minor quality loss. |
| title | DiffSoundStream: Efficient Speech Tokenization via Diffusion Decoding |
| topic | Audio and Speech Processing Machine Learning |
| url | https://arxiv.org/abs/2506.22362 |