Progressive Residual Extraction based Pre-training for Speech Representation Learning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917765800001536 |
|---|---|
| author | Wang, Tianrui Li, Jin Ma, Ziyang Cao, Rui Chen, Xie Wang, Longbiao Ge, Meng Wang, Xiaobao Wang, Yuguang Dang, Jianwu Tashi, Nyima |
| author_facet | Wang, Tianrui Li, Jin Ma, Ziyang Cao, Rui Chen, Xie Wang, Longbiao Ge, Meng Wang, Xiaobao Wang, Yuguang Dang, Jianwu Tashi, Nyima |
| contents | Self-supervised learning (SSL) has garnered significant attention in speech processing, excelling in linguistic tasks such as speech recognition. However, jointly improving the performance of pre-trained models on various downstream tasks, each requiring different speech information, poses significant challenges. To this purpose, we propose a progressive residual extraction based self-supervised learning method, named ProgRE. Specifically, we introduce two lightweight and specialized task modules into an encoder-style SSL backbone to enhance its ability to extract pitch variation and speaker information from speech. Furthermore, to prevent the interference of reinforced pitch variation and speaker information with irrelevant content information learning, we residually remove the information extracted by these two modules from the main branch. The main branch is then trained using HuBERT's speech masking prediction to ensure the performance of the Transformer's deep-layer features on content tasks. In this way, we can progressively extract pitch variation, speaker, and content representations from the input speech. Finally, we can combine multiple representations with diverse speech information using different layer weights to obtain task-specific representations for various downstream tasks. Experimental results indicate that our proposed method achieves joint performance improvements on various tasks, such as speaker identification, speech recognition, emotion recognition, speech enhancement, and voice conversion, compared to excellent SSL methods such as wav2vec2.0, HuBERT, and WavLM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_00387 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Progressive Residual Extraction based Pre-training for Speech Representation Learning Wang, Tianrui Li, Jin Ma, Ziyang Cao, Rui Chen, Xie Wang, Longbiao Ge, Meng Wang, Xiaobao Wang, Yuguang Dang, Jianwu Tashi, Nyima Audio and Speech Processing Sound Self-supervised learning (SSL) has garnered significant attention in speech processing, excelling in linguistic tasks such as speech recognition. However, jointly improving the performance of pre-trained models on various downstream tasks, each requiring different speech information, poses significant challenges. To this purpose, we propose a progressive residual extraction based self-supervised learning method, named ProgRE. Specifically, we introduce two lightweight and specialized task modules into an encoder-style SSL backbone to enhance its ability to extract pitch variation and speaker information from speech. Furthermore, to prevent the interference of reinforced pitch variation and speaker information with irrelevant content information learning, we residually remove the information extracted by these two modules from the main branch. The main branch is then trained using HuBERT's speech masking prediction to ensure the performance of the Transformer's deep-layer features on content tasks. In this way, we can progressively extract pitch variation, speaker, and content representations from the input speech. Finally, we can combine multiple representations with diverse speech information using different layer weights to obtain task-specific representations for various downstream tasks. Experimental results indicate that our proposed method achieves joint performance improvements on various tasks, such as speaker identification, speech recognition, emotion recognition, speech enhancement, and voice conversion, compared to excellent SSL methods such as wav2vec2.0, HuBERT, and WavLM. |
| title | Progressive Residual Extraction based Pre-training for Speech Representation Learning |
| topic | Audio and Speech Processing Sound |
| url | https://arxiv.org/abs/2409.00387 |