Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.02584 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908391457161216 |
|---|---|
| author | Wallbridge, Sarenne Minixhofer, Christoph Lai, Catherine Bell, Peter |
| author_facet | Wallbridge, Sarenne Minixhofer, Christoph Lai, Catherine Bell, Peter |
| contents | People exploit the predictability of lexical structures during text comprehension. Though predictable structure is also present in speech, the degree to which prosody, e.g. intonation, tempo, and loudness, contributes to such structure independently of the lexical content is unclear. This study leverages self-supervised learning (SSL) to examine the temporal granularity of structures in the acoustic correlates of prosody. Representations from our proposed Masked Prosody Model can predict perceptual labels dependent on local information, such as word boundaries, but provide the most value for labels involving longer-term structures, like emotion recognition. Probing experiments across various perceptual labels show strong relative gains over untransformed pitch, energy, and voice activity features. Our results reveal the importance of SSL training objective timescale and highlight the value of complex SSL-encoded structures compared to more constrained classical structures. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_02584 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Prosodic Structure Beyond Lexical Content: A Study of Self-Supervised Learning Wallbridge, Sarenne Minixhofer, Christoph Lai, Catherine Bell, Peter Computation and Language Artificial Intelligence Audio and Speech Processing People exploit the predictability of lexical structures during text comprehension. Though predictable structure is also present in speech, the degree to which prosody, e.g. intonation, tempo, and loudness, contributes to such structure independently of the lexical content is unclear. This study leverages self-supervised learning (SSL) to examine the temporal granularity of structures in the acoustic correlates of prosody. Representations from our proposed Masked Prosody Model can predict perceptual labels dependent on local information, such as word boundaries, but provide the most value for labels involving longer-term structures, like emotion recognition. Probing experiments across various perceptual labels show strong relative gains over untransformed pitch, energy, and voice activity features. Our results reveal the importance of SSL training objective timescale and highlight the value of complex SSL-encoded structures compared to more constrained classical structures. |
| title | Prosodic Structure Beyond Lexical Content: A Study of Self-Supervised Learning |
| topic | Computation and Language Artificial Intelligence Audio and Speech Processing |
| url | https://arxiv.org/abs/2506.02584 |