SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervision
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915694604451840 |
|---|---|
| author | Poli, Maxime Luthra, Mahi Benchekroun, Youssef Higuchi, Yosuke Gleize, Martin Shen, Jiayi Algayres, Robin Chung, Yu-An Assran, Mido Pino, Juan Dupoux, Emmanuel |
| author_facet | Poli, Maxime Luthra, Mahi Benchekroun, Youssef Higuchi, Yosuke Gleize, Martin Shen, Jiayi Algayres, Robin Chung, Yu-An Assran, Mido Pino, Juan Dupoux, Emmanuel |
| contents | The parallel advances in language modeling and speech representation learning have raised the prospect of learning language directly from speech without textual intermediates. This requires extracting semantic representations directly from speech. Our contributions are threefold. First, we introduce SpidR, a self-supervised speech representation model that efficiently learns representations with highly accessible phonetic information, which makes it particularly suited for textless spoken language modeling. It is trained on raw waveforms using a masked prediction objective combined with self-distillation and online clustering. The intermediate layers of the student model learn to predict assignments derived from the teacher's intermediate layers. This learning objective stabilizes the online clustering procedure compared to previous approaches, resulting in higher quality codebooks. SpidR outperforms wav2vec 2.0, HuBERT, WavLM, and DinoSR on downstream language modeling benchmarks (sWUGGY, sBLIMP, tSC). Second, we systematically evaluate across models and layers the correlation between speech unit quality (ABX, PNMI) and language modeling performance, validating these metrics as reliable proxies. Finally, SpidR significantly reduces pretraining time compared to HuBERT, requiring only one day of pretraining on 16 GPUs, instead of a week. This speedup is enabled by the pretraining method and an efficient codebase, which allows faster iteration and easier experimentation. We open-source the training code and model checkpoints at https://github.com/facebookresearch/spidr. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_20308 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervision Poli, Maxime Luthra, Mahi Benchekroun, Youssef Higuchi, Yosuke Gleize, Martin Shen, Jiayi Algayres, Robin Chung, Yu-An Assran, Mido Pino, Juan Dupoux, Emmanuel Computation and Language Sound Audio and Speech Processing The parallel advances in language modeling and speech representation learning have raised the prospect of learning language directly from speech without textual intermediates. This requires extracting semantic representations directly from speech. Our contributions are threefold. First, we introduce SpidR, a self-supervised speech representation model that efficiently learns representations with highly accessible phonetic information, which makes it particularly suited for textless spoken language modeling. It is trained on raw waveforms using a masked prediction objective combined with self-distillation and online clustering. The intermediate layers of the student model learn to predict assignments derived from the teacher's intermediate layers. This learning objective stabilizes the online clustering procedure compared to previous approaches, resulting in higher quality codebooks. SpidR outperforms wav2vec 2.0, HuBERT, WavLM, and DinoSR on downstream language modeling benchmarks (sWUGGY, sBLIMP, tSC). Second, we systematically evaluate across models and layers the correlation between speech unit quality (ABX, PNMI) and language modeling performance, validating these metrics as reliable proxies. Finally, SpidR significantly reduces pretraining time compared to HuBERT, requiring only one day of pretraining on 16 GPUs, instead of a week. This speedup is enabled by the pretraining method and an efficient codebase, which allows faster iteration and easier experimentation. We open-source the training code and model checkpoints at https://github.com/facebookresearch/spidr. |
| title | SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervision |
| topic | Computation and Language Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2512.20308 |