Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognition
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914653326540800 |
|---|---|
| author | Yu, Yu Yang, Chao-Han Huck Kolehmainen, Jari Shivakumar, Prashanth G. Gu, Yile Ryu, Sungho Ren, Roger Luo, Qi Gourav, Aditya Chen, I-Fan Liu, Yi-Chieh Dinh, Tuan Gandhe, Ankur Filimonov, Denis Ghosh, Shalini Stolcke, Andreas Rastow, Ariya Bulyko, Ivan |
| author_facet | Yu, Yu Yang, Chao-Han Huck Kolehmainen, Jari Shivakumar, Prashanth G. Gu, Yile Ryu, Sungho Ren, Roger Luo, Qi Gourav, Aditya Chen, I-Fan Liu, Yi-Chieh Dinh, Tuan Gandhe, Ankur Filimonov, Denis Ghosh, Shalini Stolcke, Andreas Rastow, Ariya Bulyko, Ivan |
| contents | We propose a neural language modeling system based on low-rank adaptation (LoRA) for speech recognition output rescoring. Although pretrained language models (LMs) like BERT have shown superior performance in second-pass rescoring, the high computational cost of scaling up the pretraining stage and adapting the pretrained models to specific domains limit their practical use in rescoring. Here we present a method based on low-rank decomposition to train a rescoring BERT model and adapt it to new domains using only a fraction (0.08%) of the pretrained parameters. These inserted matrices are optimized through a discriminative training objective along with a correlation-based regularization loss. The proposed low-rank adaptation Rescore-BERT (LoRB) architecture is evaluated on LibriSpeech and internal datasets with decreased training times by factors between 5.4 and 3.6. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_15223 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognition Yu, Yu Yang, Chao-Han Huck Kolehmainen, Jari Shivakumar, Prashanth G. Gu, Yile Ryu, Sungho Ren, Roger Luo, Qi Gourav, Aditya Chen, I-Fan Liu, Yi-Chieh Dinh, Tuan Gandhe, Ankur Filimonov, Denis Ghosh, Shalini Stolcke, Andreas Rastow, Ariya Bulyko, Ivan Computation and Language Artificial Intelligence Machine Learning Neural and Evolutionary Computing Sound Audio and Speech Processing We propose a neural language modeling system based on low-rank adaptation (LoRA) for speech recognition output rescoring. Although pretrained language models (LMs) like BERT have shown superior performance in second-pass rescoring, the high computational cost of scaling up the pretraining stage and adapting the pretrained models to specific domains limit their practical use in rescoring. Here we present a method based on low-rank decomposition to train a rescoring BERT model and adapt it to new domains using only a fraction (0.08%) of the pretrained parameters. These inserted matrices are optimized through a discriminative training objective along with a correlation-based regularization loss. The proposed low-rank adaptation Rescore-BERT (LoRB) architecture is evaluated on LibriSpeech and internal datasets with decreased training times by factors between 5.4 and 3.6. |
| title | Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognition |
| topic | Computation and Language Artificial Intelligence Machine Learning Neural and Evolutionary Computing Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2309.15223 |