Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2501.05310 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917315483795456 |
|---|---|
| author | Chiu, Aemon Yat Fei Fung, Kei Ching Li, Roger Tsz Yeung Li, Jingyu Lee, Tan |
| author_facet | Chiu, Aemon Yat Fei Fung, Kei Ching Li, Roger Tsz Yeung Li, Jingyu Lee, Tan |
| contents | Enhancing explainability in speech self-supervised learning (SSL) is important for developing reliable SSL-based speech processing systems. This study probes how speech SSL models encode speaker-specific information via a large-scale probing analysis of 11 models, decomposing identity into acoustic, prosodic, and paralinguistic attributes. The results confirm a general hierarchy wherein initial layers encode fundamental acoustics and middle layers synthesise abstract traits. Crucially, the consensus that final layers purely abstract linguistic content is challenged. It is discovered that larger models unexpectedly recover speaker identity in their deep layers. Furthermore, the intermediate representations of speech SSL models are found to capture dynamic prosody better than specialised speaker embeddings. These insights decode the complex internal mechanics of SSL models, providing guidelines for selecting interpretable and task-optimal representations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_05310 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Large-Scale Probing Analysis of Speaker-Specific Attributes in Self-Supervised Speech Representations Chiu, Aemon Yat Fei Fung, Kei Ching Li, Roger Tsz Yeung Li, Jingyu Lee, Tan Audio and Speech Processing Sound Enhancing explainability in speech self-supervised learning (SSL) is important for developing reliable SSL-based speech processing systems. This study probes how speech SSL models encode speaker-specific information via a large-scale probing analysis of 11 models, decomposing identity into acoustic, prosodic, and paralinguistic attributes. The results confirm a general hierarchy wherein initial layers encode fundamental acoustics and middle layers synthesise abstract traits. Crucially, the consensus that final layers purely abstract linguistic content is challenged. It is discovered that larger models unexpectedly recover speaker identity in their deep layers. Furthermore, the intermediate representations of speech SSL models are found to capture dynamic prosody better than specialised speaker embeddings. These insights decode the complex internal mechanics of SSL models, providing guidelines for selecting interpretable and task-optimal representations. |
| title | A Large-Scale Probing Analysis of Speaker-Specific Attributes in Self-Supervised Speech Representations |
| topic | Audio and Speech Processing Sound |
| url | https://arxiv.org/abs/2501.05310 |