Tracking the emergence of linguistic structure in self-supervised models learning from speech
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918425595478016 |
|---|---|
| author | Kloots, Marianne de Heer Bentum, Martijn Mohebbi, Hosein Pouw, Charlotte Shen, Gaofei Zuidema, Willem |
| author_facet | Kloots, Marianne de Heer Bentum, Martijn Mohebbi, Hosein Pouw, Charlotte Shen, Gaofei Zuidema, Willem |
| contents | Self-supervised speech models learn effective representations of spoken language, which have been shown to reflect various aspects of linguistic structure. But when does such structure emerge in model training? We study the encoding of a wide range of linguistic structures, across layers and intermediate checkpoints of six Wav2Vec2 and HuBERT models trained on spoken Dutch. We find that different levels of linguistic structure show notably distinct layerwise patterns as well as learning trajectories, which can partially be explained by differences in their degree of abstraction from the acoustic signal and the timescale at which information from the input is integrated. Moreover, we find that the level at which pre-training objectives are defined strongly affects both the layerwise organization and the learning trajectories of linguistic structures, with greater parallelism induced by higher-order prediction tasks (i.e. iteratively refined pseudo-labels). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_02043 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Tracking the emergence of linguistic structure in self-supervised models learning from speech Kloots, Marianne de Heer Bentum, Martijn Mohebbi, Hosein Pouw, Charlotte Shen, Gaofei Zuidema, Willem Computation and Language Artificial Intelligence Audio and Speech Processing Self-supervised speech models learn effective representations of spoken language, which have been shown to reflect various aspects of linguistic structure. But when does such structure emerge in model training? We study the encoding of a wide range of linguistic structures, across layers and intermediate checkpoints of six Wav2Vec2 and HuBERT models trained on spoken Dutch. We find that different levels of linguistic structure show notably distinct layerwise patterns as well as learning trajectories, which can partially be explained by differences in their degree of abstraction from the acoustic signal and the timescale at which information from the input is integrated. Moreover, we find that the level at which pre-training objectives are defined strongly affects both the layerwise organization and the learning trajectories of linguistic structures, with greater parallelism induced by higher-order prediction tasks (i.e. iteratively refined pseudo-labels). |
| title | Tracking the emergence of linguistic structure in self-supervised models learning from speech |
| topic | Computation and Language Artificial Intelligence Audio and Speech Processing |
| url | https://arxiv.org/abs/2604.02043 |