LoRA-Whisper: Parameter-Efficient and Extensible Multilingual ASR
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913384992079872 |
|---|---|
| author | Song, Zheshu Zhuo, Jianheng Yang, Yifan Ma, Ziyang Zhang, Shixiong Chen, Xie |
| author_facet | Song, Zheshu Zhuo, Jianheng Yang, Yifan Ma, Ziyang Zhang, Shixiong Chen, Xie |
| contents | Recent years have witnessed significant progress in multilingual automatic speech recognition (ASR), driven by the emergence of end-to-end (E2E) models and the scaling of multilingual datasets. Despite that, two main challenges persist in multilingual ASR: language interference and the incorporation of new languages without degrading the performance of the existing ones. This paper proposes LoRA-Whisper, which incorporates LoRA matrix into Whisper for multilingual ASR, effectively mitigating language interference. Furthermore, by leveraging LoRA and the similarities between languages, we can achieve better performance on new languages while upholding consistent performance on original ones. Experiments on a real-world task across eight languages demonstrate that our proposed LoRA-Whisper yields a relative gain of 18.5% and 23.0% over the baseline system for multilingual ASR and language expansion respectively. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_06619 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LoRA-Whisper: Parameter-Efficient and Extensible Multilingual ASR Song, Zheshu Zhuo, Jianheng Yang, Yifan Ma, Ziyang Zhang, Shixiong Chen, Xie Audio and Speech Processing Artificial Intelligence Computation and Language Recent years have witnessed significant progress in multilingual automatic speech recognition (ASR), driven by the emergence of end-to-end (E2E) models and the scaling of multilingual datasets. Despite that, two main challenges persist in multilingual ASR: language interference and the incorporation of new languages without degrading the performance of the existing ones. This paper proposes LoRA-Whisper, which incorporates LoRA matrix into Whisper for multilingual ASR, effectively mitigating language interference. Furthermore, by leveraging LoRA and the similarities between languages, we can achieve better performance on new languages while upholding consistent performance on original ones. Experiments on a real-world task across eight languages demonstrate that our proposed LoRA-Whisper yields a relative gain of 18.5% and 23.0% over the baseline system for multilingual ASR and language expansion respectively. |
| title | LoRA-Whisper: Parameter-Efficient and Extensible Multilingual ASR |
| topic | Audio and Speech Processing Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2406.06619 |