BESTOW: Efficient and Streamable Speech Language Model with the Best of Two Worlds in GPT and T5
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909233179525120 |
|---|---|
| author | Chen, Zhehuai Huang, He Hrinchuk, Oleksii Puvvada, Krishna C. Koluguri, Nithin Rao Żelasko, Piotr Balam, Jagadeesh Ginsburg, Boris |
| author_facet | Chen, Zhehuai Huang, He Hrinchuk, Oleksii Puvvada, Krishna C. Koluguri, Nithin Rao Żelasko, Piotr Balam, Jagadeesh Ginsburg, Boris |
| contents | Incorporating speech understanding capabilities into pretrained large-language models has become a vital research direction (SpeechLLM). The previous architectures can be categorized as: i) GPT-style, prepend speech prompts to the text prompts as a sequence of LLM inputs like a decoder-only model; ii) T5-style, introduce speech cross-attention to each layer of the pretrained LLMs. We propose BESTOW architecture to bring the BESt features from TwO Worlds into a single model that is highly efficient and has strong multitask capabilities. Moreover, there is no clear streaming solution for either style, especially considering the solution should generalize to speech multitask. We reformulate streamable SpeechLLM as a read-write policy problem and unifies the offline and streaming research with BESTOW architecture. Hence we demonstrate the first open-source SpeechLLM solution that enables Streaming and Multitask at scale (beyond ASR) at the same time. This streamable solution achieves very strong performance on a wide range of speech tasks (ASR, AST, SQA, unseen DynamicSuperb). It is end-to-end optimizable, with lower training/inference cost, and demonstrates LLM knowledge transferability to speech. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_19954 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | BESTOW: Efficient and Streamable Speech Language Model with the Best of Two Worlds in GPT and T5 Chen, Zhehuai Huang, He Hrinchuk, Oleksii Puvvada, Krishna C. Koluguri, Nithin Rao Żelasko, Piotr Balam, Jagadeesh Ginsburg, Boris Computation and Language Human-Computer Interaction Sound Audio and Speech Processing 68T10 I.2.7 Incorporating speech understanding capabilities into pretrained large-language models has become a vital research direction (SpeechLLM). The previous architectures can be categorized as: i) GPT-style, prepend speech prompts to the text prompts as a sequence of LLM inputs like a decoder-only model; ii) T5-style, introduce speech cross-attention to each layer of the pretrained LLMs. We propose BESTOW architecture to bring the BESt features from TwO Worlds into a single model that is highly efficient and has strong multitask capabilities. Moreover, there is no clear streaming solution for either style, especially considering the solution should generalize to speech multitask. We reformulate streamable SpeechLLM as a read-write policy problem and unifies the offline and streaming research with BESTOW architecture. Hence we demonstrate the first open-source SpeechLLM solution that enables Streaming and Multitask at scale (beyond ASR) at the same time. This streamable solution achieves very strong performance on a wide range of speech tasks (ASR, AST, SQA, unseen DynamicSuperb). It is end-to-end optimizable, with lower training/inference cost, and demonstrates LLM knowledge transferability to speech. |
| title | BESTOW: Efficient and Streamable Speech Language Model with the Best of Two Worlds in GPT and T5 |
| topic | Computation and Language Human-Computer Interaction Sound Audio and Speech Processing 68T10 I.2.7 |
| url | https://arxiv.org/abs/2406.19954 |