Towards Rehearsal-Free Multilingual ASR: A LoRA-based Case Study on Whisper
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916362895491072 |
|---|---|
| author | Xu, Tianyi Huang, Kaixun Guo, Pengcheng Zhou, Yu Huang, Longtao Xue, Hui Xie, Lei |
| author_facet | Xu, Tianyi Huang, Kaixun Guo, Pengcheng Zhou, Yu Huang, Longtao Xue, Hui Xie, Lei |
| contents | Pre-trained multilingual speech foundation models, like Whisper, have shown impressive performance across different languages. However, adapting these models to new or specific languages is computationally extensive and faces catastrophic forgetting problems. Addressing these issues, our study investigates strategies to enhance the model on new languages in the absence of original training data, while also preserving the established performance on the original languages. Specifically, we first compare various LoRA-based methods to find out their vulnerability to forgetting. To mitigate this issue, we propose to leverage the LoRA parameters from the original model for approximate orthogonal gradient descent on the new samples. Additionally, we also introduce a learnable rank coefficient to allocate trainable parameters for more efficient training. Our experiments with a Chinese Whisper model (for Uyghur and Tibetan) yield better results with a more compact parameter set. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_10680 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards Rehearsal-Free Multilingual ASR: A LoRA-based Case Study on Whisper Xu, Tianyi Huang, Kaixun Guo, Pengcheng Zhou, Yu Huang, Longtao Xue, Hui Xie, Lei Computation and Language Sound Audio and Speech Processing Pre-trained multilingual speech foundation models, like Whisper, have shown impressive performance across different languages. However, adapting these models to new or specific languages is computationally extensive and faces catastrophic forgetting problems. Addressing these issues, our study investigates strategies to enhance the model on new languages in the absence of original training data, while also preserving the established performance on the original languages. Specifically, we first compare various LoRA-based methods to find out their vulnerability to forgetting. To mitigate this issue, we propose to leverage the LoRA parameters from the original model for approximate orthogonal gradient descent on the new samples. Additionally, we also introduce a learnable rank coefficient to allocate trainable parameters for more efficient training. Our experiments with a Chinese Whisper model (for Uyghur and Tibetan) yield better results with a more compact parameter set. |
| title | Towards Rehearsal-Free Multilingual ASR: A LoRA-based Case Study on Whisper |
| topic | Computation and Language Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2408.10680 |