Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.07025 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917321310732288 |
|---|---|
| author | Gopal, Shreyas Wu, Donghang Anshul, Ashutosh Heng, Yeo Yue Peng, Yizhou Li, Haoyang Liu, Hexin Chng, Eng Siong |
| author_facet | Gopal, Shreyas Wu, Donghang Anshul, Ashutosh Heng, Yeo Yue Peng, Yizhou Li, Haoyang Liu, Hexin Chng, Eng Siong |
| contents | Speech Large Language Models (LLMs) that understand and follow instructions in many languages are useful for real-world interaction, but are difficult to train with supervised fine-tuning, requiring large, task-specific speech corpora. While recent distillation-based approaches train performant English-only Speech LLMs using only annotated ASR data by aligning text and speech using only a lightweight projector, these models under-perform when scaled to multilingual settings due to language interference in the shared projector. We address this by introducing language-aware distillation using a query bank and a gating network that selects or mixes query tokens using a Q-Former projector. Our approach shows gains of 14% over matched multilingual distillation baselines on instruction following. We further synthesize Audio-MLQA, a multilingual spoken QA benchmark built on MLQA with high-quality TTS questions. Our best model improves over existing Speech LLM baselines by 32% on Audio-MLQA. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_07025 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervision Gopal, Shreyas Wu, Donghang Anshul, Ashutosh Heng, Yeo Yue Peng, Yizhou Li, Haoyang Liu, Hexin Chng, Eng Siong Computation and Language Speech Large Language Models (LLMs) that understand and follow instructions in many languages are useful for real-world interaction, but are difficult to train with supervised fine-tuning, requiring large, task-specific speech corpora. While recent distillation-based approaches train performant English-only Speech LLMs using only annotated ASR data by aligning text and speech using only a lightweight projector, these models under-perform when scaled to multilingual settings due to language interference in the shared projector. We address this by introducing language-aware distillation using a query bank and a gating network that selects or mixes query tokens using a Q-Former projector. Our approach shows gains of 14% over matched multilingual distillation baselines on instruction following. We further synthesize Audio-MLQA, a multilingual spoken QA benchmark built on MLQA with high-quality TTS questions. Our best model improves over existing Speech LLM baselines by 32% on Audio-MLQA. |
| title | Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervision |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2603.07025 |