SALSA: Speedy ASR-LLM Synchronous Aggregation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909300449869824 |
|---|---|
| author | Mittal, Ashish Prabhu, Darshan Sarawagi, Sunita Jyothi, Preethi |
| author_facet | Mittal, Ashish Prabhu, Darshan Sarawagi, Sunita Jyothi, Preethi |
| contents | Harnessing pre-trained LLMs to improve ASR systems, particularly for low-resource languages, is now an emerging area of research. Existing methods range from using LLMs for ASR error correction to tightly coupled systems that replace the ASR decoder with the LLM. These approaches either increase decoding time or require expensive training of the cross-attention layers. We propose SALSA, which couples the decoder layers of the ASR to the LLM decoder, while synchronously advancing both decoders. Such coupling is performed with a simple projection of the last decoder state, and is thus significantly more training efficient than earlier approaches. A challenge of our proposed coupling is handling the mismatch between the tokenizers of the LLM and ASR systems. We handle this mismatch using cascading tokenization with respect to the LLM and ASR vocabularies. We evaluate SALSA on 8 low-resource languages in the FLEURS benchmark, yielding substantial WER reductions of up to 38%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_16542 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SALSA: Speedy ASR-LLM Synchronous Aggregation Mittal, Ashish Prabhu, Darshan Sarawagi, Sunita Jyothi, Preethi Computation and Language Machine Learning Sound Audio and Speech Processing Harnessing pre-trained LLMs to improve ASR systems, particularly for low-resource languages, is now an emerging area of research. Existing methods range from using LLMs for ASR error correction to tightly coupled systems that replace the ASR decoder with the LLM. These approaches either increase decoding time or require expensive training of the cross-attention layers. We propose SALSA, which couples the decoder layers of the ASR to the LLM decoder, while synchronously advancing both decoders. Such coupling is performed with a simple projection of the last decoder state, and is thus significantly more training efficient than earlier approaches. A challenge of our proposed coupling is handling the mismatch between the tokenizers of the LLM and ASR systems. We handle this mismatch using cascading tokenization with respect to the LLM and ASR vocabularies. We evaluate SALSA on 8 low-resource languages in the FLEURS benchmark, yielding substantial WER reductions of up to 38%. |
| title | SALSA: Speedy ASR-LLM Synchronous Aggregation |
| topic | Computation and Language Machine Learning Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2408.16542 |