SSHR: Leveraging Self-supervised Hierarchical Representations for Multilingual Automatic Speech Recognition
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929329178411008 |
|---|---|
| author | Xue, Hongfei Shao, Qijie Huang, Kaixun Chen, Peikun Liu, Jie Xie, Lei |
| author_facet | Xue, Hongfei Shao, Qijie Huang, Kaixun Chen, Peikun Liu, Jie Xie, Lei |
| contents | Multilingual automatic speech recognition (ASR) systems have garnered attention for their potential to extend language coverage globally. While self-supervised learning (SSL) models, like MMS, have demonstrated their effectiveness in multilingual ASR, it is worth noting that various layers' representations potentially contain distinct information that has not been fully leveraged. In this study, we propose a novel method that leverages self-supervised hierarchical representations (SSHR) to fine-tune the MMS model. We first analyze the different layers of MMS and show that the middle layers capture language-related information, and the high layers encode content-related information, which gradually decreases in the final layers. Then, we extract a language-related frame from correlated middle layers and guide specific language extraction through self-attention mechanisms. Additionally, we steer the model toward acquiring more content-related information in the final layers using our proposed Cross-CTC. We evaluate SSHR on two multilingual datasets, Common Voice and ML-SUPERB, and the experimental results demonstrate that our method achieves state-of-the-art performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_16937 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | SSHR: Leveraging Self-supervised Hierarchical Representations for Multilingual Automatic Speech Recognition Xue, Hongfei Shao, Qijie Huang, Kaixun Chen, Peikun Liu, Jie Xie, Lei Computation and Language Sound Audio and Speech Processing Multilingual automatic speech recognition (ASR) systems have garnered attention for their potential to extend language coverage globally. While self-supervised learning (SSL) models, like MMS, have demonstrated their effectiveness in multilingual ASR, it is worth noting that various layers' representations potentially contain distinct information that has not been fully leveraged. In this study, we propose a novel method that leverages self-supervised hierarchical representations (SSHR) to fine-tune the MMS model. We first analyze the different layers of MMS and show that the middle layers capture language-related information, and the high layers encode content-related information, which gradually decreases in the final layers. Then, we extract a language-related frame from correlated middle layers and guide specific language extraction through self-attention mechanisms. Additionally, we steer the model toward acquiring more content-related information in the final layers using our proposed Cross-CTC. We evaluate SSHR on two multilingual datasets, Common Voice and ML-SUPERB, and the experimental results demonstrate that our method achieves state-of-the-art performance. |
| title | SSHR: Leveraging Self-supervised Hierarchical Representations for Multilingual Automatic Speech Recognition |
| topic | Computation and Language Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2309.16937 |