SD-HuBERT: Sentence-Level Self-Distillation Induces Syllabic Organization in HuBERT
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917981584359424 |
|---|---|
| author | Cho, Cheol Jun Mohamed, Abdelrahman Li, Shang-Wen Black, Alan W Anumanchipalli, Gopala K. |
| author_facet | Cho, Cheol Jun Mohamed, Abdelrahman Li, Shang-Wen Black, Alan W Anumanchipalli, Gopala K. |
| contents | Data-driven unit discovery in self-supervised learning (SSL) of speech has embarked on a new era of spoken language processing. Yet, the discovered units often remain in phonetic space and the units beyond phonemes are largely underexplored. Here, we demonstrate that a syllabic organization emerges in learning sentence-level representation of speech. In particular, we adopt "self-distillation" objective to fine-tune the pretrained HuBERT with an aggregator token that summarizes the entire sentence. Without any supervision, the resulting model draws definite boundaries in speech, and the representations across frames exhibit salient syllabic structures. We demonstrate that this emergent structure largely corresponds to the ground truth syllables. Furthermore, we propose a new benchmark task, Spoken Speech ABX, for evaluating sentence-level representation of speech. When compared to previous models, our model outperforms in both unsupervised syllable discovery and learning sentence-level representation. Together, we demonstrate that the self-distillation of HuBERT gives rise to syllabic organization without relying on external labels or modalities, and potentially provides novel data-driven units for spoken language modeling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_10803 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | SD-HuBERT: Sentence-Level Self-Distillation Induces Syllabic Organization in HuBERT Cho, Cheol Jun Mohamed, Abdelrahman Li, Shang-Wen Black, Alan W Anumanchipalli, Gopala K. Computation and Language Audio and Speech Processing Data-driven unit discovery in self-supervised learning (SSL) of speech has embarked on a new era of spoken language processing. Yet, the discovered units often remain in phonetic space and the units beyond phonemes are largely underexplored. Here, we demonstrate that a syllabic organization emerges in learning sentence-level representation of speech. In particular, we adopt "self-distillation" objective to fine-tune the pretrained HuBERT with an aggregator token that summarizes the entire sentence. Without any supervision, the resulting model draws definite boundaries in speech, and the representations across frames exhibit salient syllabic structures. We demonstrate that this emergent structure largely corresponds to the ground truth syllables. Furthermore, we propose a new benchmark task, Spoken Speech ABX, for evaluating sentence-level representation of speech. When compared to previous models, our model outperforms in both unsupervised syllable discovery and learning sentence-level representation. Together, we demonstrate that the self-distillation of HuBERT gives rise to syllabic organization without relying on external labels or modalities, and potentially provides novel data-driven units for spoken language modeling. |
| title | SD-HuBERT: Sentence-Level Self-Distillation Induces Syllabic Organization in HuBERT |
| topic | Computation and Language Audio and Speech Processing |
| url | https://arxiv.org/abs/2310.10803 |