Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.11086 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910833790943232 |
|---|---|
| author | Yadav, Hemant Shah, Rajiv Ratn Sitaram, Sunayana |
| author_facet | Yadav, Hemant Shah, Rajiv Ratn Sitaram, Sunayana |
| contents | Information in speech can be categorized into two groups: Content (what is being said, such as linguistics) and Other (how it is expressed such as information about speaker and paralinguistic features). Current self-supervised learning (SSL) methods are shown to divide the model's representational-depth or layers in two, with earlier layers specializing in Other and later layers in Content related tasks. This layer-wise division is inherently sub-optimal, as neither information type can use all layers to build hierarchical representations. To address this, we propose JOOCI, a novel speech representation learning method that does not compromise on the representational-depth for either information type. JOOCI outperforms WavLM by 26.5%, and other models of similar size (100M parameters), when evaluated on two speaker recognition and two language tasks from the SUPERB benchmark, demonstrating its effectiveness in Jointly Optimizing Other and Content Information (JOOCI). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_11086 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | JOOCI: a Framework for Learning Comprehensive Speech Representations Yadav, Hemant Shah, Rajiv Ratn Sitaram, Sunayana Computation and Language Information in speech can be categorized into two groups: Content (what is being said, such as linguistics) and Other (how it is expressed such as information about speaker and paralinguistic features). Current self-supervised learning (SSL) methods are shown to divide the model's representational-depth or layers in two, with earlier layers specializing in Other and later layers in Content related tasks. This layer-wise division is inherently sub-optimal, as neither information type can use all layers to build hierarchical representations. To address this, we propose JOOCI, a novel speech representation learning method that does not compromise on the representational-depth for either information type. JOOCI outperforms WavLM by 26.5%, and other models of similar size (100M parameters), when evaluated on two speaker recognition and two language tasks from the SUPERB benchmark, demonstrating its effectiveness in Jointly Optimizing Other and Content Information (JOOCI). |
| title | JOOCI: a Framework for Learning Comprehensive Speech Representations |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.11086 |