SIFT-50M: A Large-Scale Multilingual Dataset for Speech Instruction Fine-Tuning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917988428414976 |
|---|---|
| author | Pandey, Prabhat Swaminathan, Rupak Vignesh Girish, K V Vijay Sen, Arunasish Xie, Jian Strimel, Grant P. Schwarz, Andreas |
| author_facet | Pandey, Prabhat Swaminathan, Rupak Vignesh Girish, K V Vijay Sen, Arunasish Xie, Jian Strimel, Grant P. Schwarz, Andreas |
| contents | We introduce SIFT (Speech Instruction Fine-Tuning), a 50M-example dataset designed for instruction fine-tuning and pre-training of speech-text large language models (LLMs). SIFT-50M is built from publicly available speech corpora, which collectively contain 14K hours of speech, and leverages LLMs along with off-the-shelf expert models. The dataset spans five languages, encompassing a diverse range of speech understanding as well as controllable speech generation instructions. Using SIFT-50M, we train SIFT-LLM, which outperforms existing speech-text LLMs on instruction-following benchmarks while achieving competitive performance on foundational speech tasks. To support further research, we also introduce EvalSIFT, a benchmark dataset specifically designed to evaluate the instruction-following capabilities of speech-text LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_09081 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SIFT-50M: A Large-Scale Multilingual Dataset for Speech Instruction Fine-Tuning Pandey, Prabhat Swaminathan, Rupak Vignesh Girish, K V Vijay Sen, Arunasish Xie, Jian Strimel, Grant P. Schwarz, Andreas Audio and Speech Processing Artificial Intelligence Computation and Language We introduce SIFT (Speech Instruction Fine-Tuning), a 50M-example dataset designed for instruction fine-tuning and pre-training of speech-text large language models (LLMs). SIFT-50M is built from publicly available speech corpora, which collectively contain 14K hours of speech, and leverages LLMs along with off-the-shelf expert models. The dataset spans five languages, encompassing a diverse range of speech understanding as well as controllable speech generation instructions. Using SIFT-50M, we train SIFT-LLM, which outperforms existing speech-text LLMs on instruction-following benchmarks while achieving competitive performance on foundational speech tasks. To support further research, we also introduce EvalSIFT, a benchmark dataset specifically designed to evaluate the instruction-following capabilities of speech-text LLMs. |
| title | SIFT-50M: A Large-Scale Multilingual Dataset for Speech Instruction Fine-Tuning |
| topic | Audio and Speech Processing Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2504.09081 |