Distilling an End-to-End Voice Assistant Without Instruction Training Data
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910631308820480 |
|---|---|
| author | Held, William Li, Ella Ryan, Michael Shi, Weiyan Zhang, Yanzhe Yang, Diyi |
| author_facet | Held, William Li, Ella Ryan, Michael Shi, Weiyan Zhang, Yanzhe Yang, Diyi |
| contents | Voice assistants, such as Siri and Google Assistant, typically model audio and text separately, resulting in lost speech information and increased complexity. Recent efforts to address this with end-to-end Speech Large Language Models (LLMs) trained with supervised finetuning (SFT)
have led to models ``forgetting" capabilities from text-only LLMs. Our work proposes an alternative paradigm for training Speech LLMs without instruction data, using the response of a text-only LLM to transcripts as self-supervision. Importantly, this process can be performed without annotated responses. We show that our Distilled Voice Assistant (DiVA) generalizes to Spoken Question Answering, Classification, and Translation. Furthermore, we show that DiVA better meets user preferences, achieving a 72\% win rate compared with state-of-the-art models like Qwen 2 Audio, despite using $>$100x less training compute. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_02678 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Distilling an End-to-End Voice Assistant Without Instruction Training Data Held, William Li, Ella Ryan, Michael Shi, Weiyan Zhang, Yanzhe Yang, Diyi Computation and Language Artificial Intelligence Voice assistants, such as Siri and Google Assistant, typically model audio and text separately, resulting in lost speech information and increased complexity. Recent efforts to address this with end-to-end Speech Large Language Models (LLMs) trained with supervised finetuning (SFT) have led to models ``forgetting" capabilities from text-only LLMs. Our work proposes an alternative paradigm for training Speech LLMs without instruction data, using the response of a text-only LLM to transcripts as self-supervision. Importantly, this process can be performed without annotated responses. We show that our Distilled Voice Assistant (DiVA) generalizes to Spoken Question Answering, Classification, and Translation. Furthermore, we show that DiVA better meets user preferences, achieving a 72\% win rate compared with state-of-the-art models like Qwen 2 Audio, despite using $>$100x less training compute. |
| title | Distilling an End-to-End Voice Assistant Without Instruction Training Data |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2410.02678 |