Investigating Training Strategies and Model Robustness of Low-Rank Adaptation for Language Modeling in Speech Recognition
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916097888878592 |
|---|---|
| author | Yu, Yu Yang, Chao-Han Huck Dinh, Tuan Ryu, Sungho Kolehmainen, Jari Ren, Roger Filimonov, Denis Shivakumar, Prashanth G. Gandhe, Ankur Rastow, Ariya Xu, Jia Bulyko, Ivan Stolcke, Andreas |
| author_facet | Yu, Yu Yang, Chao-Han Huck Dinh, Tuan Ryu, Sungho Kolehmainen, Jari Ren, Roger Filimonov, Denis Shivakumar, Prashanth G. Gandhe, Ankur Rastow, Ariya Xu, Jia Bulyko, Ivan Stolcke, Andreas |
| contents | The use of low-rank adaptation (LoRA) with frozen pretrained language models (PLMs) has become increasing popular as a mainstream, resource-efficient modeling approach for memory-constrained hardware. In this study, we first explore how to enhance model performance by introducing various LoRA training strategies, achieving relative word error rate reductions of 3.50\% on the public Librispeech dataset and of 3.67\% on an internal dataset in the messaging domain. To further characterize the stability of LoRA-based second-pass speech recognition models, we examine robustness against input perturbations. These perturbations are rooted in homophone replacements and a novel metric called N-best Perturbation-based Rescoring Robustness (NPRR), both designed to measure the relative degradation in the performance of rescoring models. Our experimental results indicate that while advanced variants of LoRA, such as dynamic rank-allocated LoRA, lead to performance degradation in $1$-best perturbation, they alleviate the degradation in $N$-best perturbation. This finding is in comparison to fully-tuned models and vanilla LoRA tuning baselines, suggesting that a comprehensive selection is needed when using LoRA-based adaptation for compute-cost savings and robust language modeling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_10447 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Investigating Training Strategies and Model Robustness of Low-Rank Adaptation for Language Modeling in Speech Recognition Yu, Yu Yang, Chao-Han Huck Dinh, Tuan Ryu, Sungho Kolehmainen, Jari Ren, Roger Filimonov, Denis Shivakumar, Prashanth G. Gandhe, Ankur Rastow, Ariya Xu, Jia Bulyko, Ivan Stolcke, Andreas Computation and Language Artificial Intelligence Machine Learning Neural and Evolutionary Computing Sound Audio and Speech Processing The use of low-rank adaptation (LoRA) with frozen pretrained language models (PLMs) has become increasing popular as a mainstream, resource-efficient modeling approach for memory-constrained hardware. In this study, we first explore how to enhance model performance by introducing various LoRA training strategies, achieving relative word error rate reductions of 3.50\% on the public Librispeech dataset and of 3.67\% on an internal dataset in the messaging domain. To further characterize the stability of LoRA-based second-pass speech recognition models, we examine robustness against input perturbations. These perturbations are rooted in homophone replacements and a novel metric called N-best Perturbation-based Rescoring Robustness (NPRR), both designed to measure the relative degradation in the performance of rescoring models. Our experimental results indicate that while advanced variants of LoRA, such as dynamic rank-allocated LoRA, lead to performance degradation in $1$-best perturbation, they alleviate the degradation in $N$-best perturbation. This finding is in comparison to fully-tuned models and vanilla LoRA tuning baselines, suggesting that a comprehensive selection is needed when using LoRA-based adaptation for compute-cost savings and robust language modeling. |
| title | Investigating Training Strategies and Model Robustness of Low-Rank Adaptation for Language Modeling in Speech Recognition |
| topic | Computation and Language Artificial Intelligence Machine Learning Neural and Evolutionary Computing Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2401.10447 |