Efficient Streaming LLM for Speech Recognition
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910634394779648 |
|---|---|
| author | Jia, Junteng Keren, Gil Zhou, Wei Lakomkin, Egor Zhang, Xiaohui Wu, Chunyang Seide, Frank Mahadeokar, Jay Kalinli, Ozlem |
| author_facet | Jia, Junteng Keren, Gil Zhou, Wei Lakomkin, Egor Zhang, Xiaohui Wu, Chunyang Seide, Frank Mahadeokar, Jay Kalinli, Ozlem |
| contents | Recent works have shown that prompting large language models with audio encodings can unlock speech recognition capabilities. However, existing techniques do not scale efficiently, especially while handling long form streaming audio inputs -- not only do they extrapolate poorly beyond the audio length seen during training, but they are also computationally inefficient due to the quadratic cost of attention.
In this work, we introduce SpeechLLM-XL, a linear scaling decoder-only model for streaming speech recognition. We process audios in configurable chunks using limited attention window for reduced computation, and the text tokens for each audio chunk are generated auto-regressively until an EOS is predicted. During training, the transcript is segmented into chunks, using a CTC forced alignment estimated from encoder output. SpeechLLM-XL with 1.28 seconds chunk size achieves 2.7%/6.7% WER on LibriSpeech test clean/other, and it shows no quality degradation on long form utterances 10x longer than the training utterances. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_03752 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Efficient Streaming LLM for Speech Recognition Jia, Junteng Keren, Gil Zhou, Wei Lakomkin, Egor Zhang, Xiaohui Wu, Chunyang Seide, Frank Mahadeokar, Jay Kalinli, Ozlem Sound Artificial Intelligence Computation and Language Audio and Speech Processing Recent works have shown that prompting large language models with audio encodings can unlock speech recognition capabilities. However, existing techniques do not scale efficiently, especially while handling long form streaming audio inputs -- not only do they extrapolate poorly beyond the audio length seen during training, but they are also computationally inefficient due to the quadratic cost of attention. In this work, we introduce SpeechLLM-XL, a linear scaling decoder-only model for streaming speech recognition. We process audios in configurable chunks using limited attention window for reduced computation, and the text tokens for each audio chunk are generated auto-regressively until an EOS is predicted. During training, the transcript is segmented into chunks, using a CTC forced alignment estimated from encoder output. SpeechLLM-XL with 1.28 seconds chunk size achieves 2.7%/6.7% WER on LibriSpeech test clean/other, and it shows no quality degradation on long form utterances 10x longer than the training utterances. |
| title | Efficient Streaming LLM for Speech Recognition |
| topic | Sound Artificial Intelligence Computation and Language Audio and Speech Processing |
| url | https://arxiv.org/abs/2410.03752 |