Transfer Learning of Transformer-based Speech Recognition Models from Czech to Slovak
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913444100308992 |
|---|---|
| author | Lehečka, Jan Psutka, Josef V. Psutka, Josef |
| author_facet | Lehečka, Jan Psutka, Josef V. Psutka, Josef |
| contents | In this paper, we are comparing several methods of training the Slovak speech recognition models based on the Transformers architecture. Specifically, we are exploring the approach of transfer learning from the existing Czech pre-trained Wav2Vec 2.0 model into Slovak. We are demonstrating the benefits of the proposed approach on three Slovak datasets. Our Slovak models scored the best results when initializing the weights from the Czech model at the beginning of the pre-training phase. Our results show that the knowledge stored in the Cezch pre-trained model can be successfully reused to solve tasks in Slovak while outperforming even much larger public multilingual models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2306_04399 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Transfer Learning of Transformer-based Speech Recognition Models from Czech to Slovak Lehečka, Jan Psutka, Josef V. Psutka, Josef Computation and Language In this paper, we are comparing several methods of training the Slovak speech recognition models based on the Transformers architecture. Specifically, we are exploring the approach of transfer learning from the existing Czech pre-trained Wav2Vec 2.0 model into Slovak. We are demonstrating the benefits of the proposed approach on three Slovak datasets. Our Slovak models scored the best results when initializing the weights from the Czech model at the beginning of the pre-training phase. Our results show that the knowledge stored in the Cezch pre-trained model can be successfully reused to solve tasks in Slovak while outperforming even much larger public multilingual models. |
| title | Transfer Learning of Transformer-based Speech Recognition Models from Czech to Slovak |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2306.04399 |