Scaling Speech Tokenizers with Diffusion Autoencoders
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915779850534912 |
|---|---|
| author | Wang, Yuancheng Tang, Zhenyu Wang, Yun Hinsvark, Arthur Liu, Yingru Li, Yinghao Peng, Kainan Ao, Junyi Ma, Mingbo Seltzer, Mike He, Qing Liu, Xubo |
| author_facet | Wang, Yuancheng Tang, Zhenyu Wang, Yun Hinsvark, Arthur Liu, Yingru Li, Yinghao Peng, Kainan Ao, Junyi Ma, Mingbo Seltzer, Mike He, Qing Liu, Xubo |
| contents | Speech tokenizers are foundational to speech language models, yet existing approaches face two major challenges: (1) balancing trade-offs between encoding semantics for understanding and acoustics for reconstruction, and (2) achieving low bit rates and low token rates. We propose Speech Diffusion Tokenizer (SiTok), a diffusion autoencoder that jointly learns semantic-rich representations through supervised learning and enables high-fidelity audio reconstruction with diffusion. We scale SiTok to 1.6B parameters and train it on 2 million hours of speech. Experiments show that SiTok outperforms strong baselines on understanding, reconstruction and generation tasks, at an extremely low token rate of $12.5$ Hz and a bit-rate of 200 bits-per-second. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_06602 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Scaling Speech Tokenizers with Diffusion Autoencoders Wang, Yuancheng Tang, Zhenyu Wang, Yun Hinsvark, Arthur Liu, Yingru Li, Yinghao Peng, Kainan Ao, Junyi Ma, Mingbo Seltzer, Mike He, Qing Liu, Xubo Sound Artificial Intelligence Machine Learning Audio and Speech Processing Speech tokenizers are foundational to speech language models, yet existing approaches face two major challenges: (1) balancing trade-offs between encoding semantics for understanding and acoustics for reconstruction, and (2) achieving low bit rates and low token rates. We propose Speech Diffusion Tokenizer (SiTok), a diffusion autoencoder that jointly learns semantic-rich representations through supervised learning and enables high-fidelity audio reconstruction with diffusion. We scale SiTok to 1.6B parameters and train it on 2 million hours of speech. Experiments show that SiTok outperforms strong baselines on understanding, reconstruction and generation tasks, at an extremely low token rate of $12.5$ Hz and a bit-rate of 200 bits-per-second. |
| title | Scaling Speech Tokenizers with Diffusion Autoencoders |
| topic | Sound Artificial Intelligence Machine Learning Audio and Speech Processing |
| url | https://arxiv.org/abs/2602.06602 |