SpeechVerse: A Large-scale Generalizable Audio Language Model
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913755326054400 |
|---|---|
| author | Das, Nilaksh Dingliwal, Saket Ronanki, Srikanth Paturi, Rohit Huang, Zhaocheng Mathur, Prashant Yuan, Jie Bekal, Dhanush Niu, Xing Jayanthi, Sai Muralidhar Li, Xilai Mundnich, Karel Sunkara, Monica Bodapati, Sravan Srinivasan, Sundararajan Han, Kyu J Kirchhoff, Katrin |
| author_facet | Das, Nilaksh Dingliwal, Saket Ronanki, Srikanth Paturi, Rohit Huang, Zhaocheng Mathur, Prashant Yuan, Jie Bekal, Dhanush Niu, Xing Jayanthi, Sai Muralidhar Li, Xilai Mundnich, Karel Sunkara, Monica Bodapati, Sravan Srinivasan, Sundararajan Han, Kyu J Kirchhoff, Katrin |
| contents | Large language models (LLMs) have shown incredible proficiency in performing tasks that require semantic understanding of natural language instructions. Recently, many works have further expanded this capability to perceive multimodal audio and text inputs, but their capabilities are often limited to specific fine-tuned tasks such as automatic speech recognition and translation. We therefore develop SpeechVerse, a robust multi-task training and curriculum learning framework that combines pre-trained speech and text foundation models via a small set of learnable parameters, while keeping the pre-trained models frozen during training. The models are instruction finetuned using continuous latent representations extracted from the speech foundation model to achieve optimal zero-shot performance on a diverse range of speech processing tasks using natural language instructions. We perform extensive benchmarking that includes comparing our model performance against traditional baselines across several datasets and tasks. Furthermore, we evaluate the model's capability for generalized instruction following by testing on out-of-domain datasets, novel prompts, and unseen tasks. Our empirical experiments reveal that our multi-task SpeechVerse model is even superior to conventional task-specific baselines on 9 out of the 11 tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_08295 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SpeechVerse: A Large-scale Generalizable Audio Language Model Das, Nilaksh Dingliwal, Saket Ronanki, Srikanth Paturi, Rohit Huang, Zhaocheng Mathur, Prashant Yuan, Jie Bekal, Dhanush Niu, Xing Jayanthi, Sai Muralidhar Li, Xilai Mundnich, Karel Sunkara, Monica Bodapati, Sravan Srinivasan, Sundararajan Han, Kyu J Kirchhoff, Katrin Computation and Language Sound Audio and Speech Processing Large language models (LLMs) have shown incredible proficiency in performing tasks that require semantic understanding of natural language instructions. Recently, many works have further expanded this capability to perceive multimodal audio and text inputs, but their capabilities are often limited to specific fine-tuned tasks such as automatic speech recognition and translation. We therefore develop SpeechVerse, a robust multi-task training and curriculum learning framework that combines pre-trained speech and text foundation models via a small set of learnable parameters, while keeping the pre-trained models frozen during training. The models are instruction finetuned using continuous latent representations extracted from the speech foundation model to achieve optimal zero-shot performance on a diverse range of speech processing tasks using natural language instructions. We perform extensive benchmarking that includes comparing our model performance against traditional baselines across several datasets and tasks. Furthermore, we evaluate the model's capability for generalized instruction following by testing on out-of-domain datasets, novel prompts, and unseen tasks. Our empirical experiments reveal that our multi-task SpeechVerse model is even superior to conventional task-specific baselines on 9 out of the 11 tasks. |
| title | SpeechVerse: A Large-scale Generalizable Audio Language Model |
| topic | Computation and Language Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2405.08295 |