XLSR-Transducer: Streaming ASR for Self-Supervised Pretrained Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910640250028032 |
|---|---|
| author | Kumar, Shashi Madikeri, Srikanth Zuluaga-Gomez, Juan Villatoro-Tello, Esaú Thorbecke, Iuliia Motlicek, Petr E, Manjunath K Ganapathiraju, Aravind |
| author_facet | Kumar, Shashi Madikeri, Srikanth Zuluaga-Gomez, Juan Villatoro-Tello, Esaú Thorbecke, Iuliia Motlicek, Petr E, Manjunath K Ganapathiraju, Aravind |
| contents | Self-supervised pretrained models exhibit competitive performance in automatic speech recognition on finetuning, even with limited in-domain supervised data. However, popular pretrained models are not suitable for streaming ASR because they are trained with full attention context. In this paper, we introduce XLSR-Transducer, where the XLSR-53 model is used as encoder in transducer setup. Our experiments on the AMI dataset reveal that the XLSR-Transducer achieves 4% absolute WER improvement over Whisper large-v2 and 8% over a Zipformer transducer model trained from scratch. To enable streaming capabilities, we investigate different attention masking patterns in the self-attention computation of transformer layers within the XLSR-53 model. We validate XLSR-Transducer on AMI and 5 languages from CommonVoice under low-resource scenarios. Finally, with the introduction of attention sinks, we reduce the left context by half while achieving a relative 12% improvement in WER. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_04439 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | XLSR-Transducer: Streaming ASR for Self-Supervised Pretrained Models Kumar, Shashi Madikeri, Srikanth Zuluaga-Gomez, Juan Villatoro-Tello, Esaú Thorbecke, Iuliia Motlicek, Petr E, Manjunath K Ganapathiraju, Aravind Audio and Speech Processing Self-supervised pretrained models exhibit competitive performance in automatic speech recognition on finetuning, even with limited in-domain supervised data. However, popular pretrained models are not suitable for streaming ASR because they are trained with full attention context. In this paper, we introduce XLSR-Transducer, where the XLSR-53 model is used as encoder in transducer setup. Our experiments on the AMI dataset reveal that the XLSR-Transducer achieves 4% absolute WER improvement over Whisper large-v2 and 8% over a Zipformer transducer model trained from scratch. To enable streaming capabilities, we investigate different attention masking patterns in the self-attention computation of transformer layers within the XLSR-53 model. We validate XLSR-Transducer on AMI and 5 languages from CommonVoice under low-resource scenarios. Finally, with the introduction of attention sinks, we reduce the left context by half while achieving a relative 12% improvement in WER. |
| title | XLSR-Transducer: Streaming ASR for Self-Supervised Pretrained Models |
| topic | Audio and Speech Processing |
| url | https://arxiv.org/abs/2407.04439 |