Speech Prefix-Tuning with RNNT Loss for Improving LLM Predictions
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929393401593856 |
|---|---|
| author | Baskar, Murali Karthick Rosenberg, Andrew Ramabhadran, Bhuvana Gaur, Neeraj Meng, Zhong |
| author_facet | Baskar, Murali Karthick Rosenberg, Andrew Ramabhadran, Bhuvana Gaur, Neeraj Meng, Zhong |
| contents | In this paper, we focus on addressing the constraints faced when applying LLMs to ASR. Recent works utilize prefixLM-type models, which directly apply speech as a prefix to LLMs for ASR. We have found that optimizing speech prefixes leads to better ASR performance and propose applying RNNT loss to perform speech prefix-tuning. This is a simple approach and does not increase the model complexity or alter the inference pipeline. We also propose language-based soft prompting to further improve with frozen LLMs. Empirical analysis on realtime testset from 10 Indic languages demonstrate that our proposed speech prefix-tuning yields improvements with both frozen and fine-tuned LLMs. Our recognition results on an average of 10 Indics show that the proposed prefix-tuning with RNNT loss results in a 12\% relative improvement in WER over the baseline with a fine-tuned LLM. Our proposed approches with the frozen LLM leads to a 31\% relative improvement over basic soft-prompting prefixLM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_14701 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Speech Prefix-Tuning with RNNT Loss for Improving LLM Predictions Baskar, Murali Karthick Rosenberg, Andrew Ramabhadran, Bhuvana Gaur, Neeraj Meng, Zhong Artificial Intelligence Computation and Language Sound Audio and Speech Processing In this paper, we focus on addressing the constraints faced when applying LLMs to ASR. Recent works utilize prefixLM-type models, which directly apply speech as a prefix to LLMs for ASR. We have found that optimizing speech prefixes leads to better ASR performance and propose applying RNNT loss to perform speech prefix-tuning. This is a simple approach and does not increase the model complexity or alter the inference pipeline. We also propose language-based soft prompting to further improve with frozen LLMs. Empirical analysis on realtime testset from 10 Indic languages demonstrate that our proposed speech prefix-tuning yields improvements with both frozen and fine-tuned LLMs. Our recognition results on an average of 10 Indics show that the proposed prefix-tuning with RNNT loss results in a 12\% relative improvement in WER over the baseline with a fine-tuned LLM. Our proposed approches with the frozen LLM leads to a 31\% relative improvement over basic soft-prompting prefixLM. |
| title | Speech Prefix-Tuning with RNNT Loss for Improving LLM Predictions |
| topic | Artificial Intelligence Computation and Language Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2406.14701 |