WHISMA: A Speech-LLM to Perform Zero-shot Spoken Language Understanding
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910581916696576 |
|---|---|
| author | Li, Mohan Do, Cong-Thanh Keizer, Simon Farag, Youmna Stoyanchev, Svetlana Doddipatla, Rama |
| author_facet | Li, Mohan Do, Cong-Thanh Keizer, Simon Farag, Youmna Stoyanchev, Svetlana Doddipatla, Rama |
| contents | Speech large language models (speech-LLMs) integrate speech and text-based foundation models to provide a unified framework for handling a wide range of downstream tasks. In this paper, we introduce WHISMA, a speech-LLM tailored for spoken language understanding (SLU) that demonstrates robust performance in various zero-shot settings. WHISMA combines the speech encoder from Whisper with the Llama-3 LLM, and is fine-tuned in a parameter-efficient manner on a comprehensive collection of SLU-related datasets. Our experiments show that WHISMA significantly improves the zero-shot slot filling performance on the SLURP benchmark, achieving a relative gain of 26.6% compared to the current state-of-the-art model. Furthermore, to evaluate WHISMA's generalisation capabilities to unseen domains, we develop a new task-agnostic benchmark named SLU-GLUE. The evaluation results indicate that WHISMA outperforms an existing speech-LLM (Qwen-Audio) with a relative gain of 33.0%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_16423 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | WHISMA: A Speech-LLM to Perform Zero-shot Spoken Language Understanding Li, Mohan Do, Cong-Thanh Keizer, Simon Farag, Youmna Stoyanchev, Svetlana Doddipatla, Rama Audio and Speech Processing Sound Speech large language models (speech-LLMs) integrate speech and text-based foundation models to provide a unified framework for handling a wide range of downstream tasks. In this paper, we introduce WHISMA, a speech-LLM tailored for spoken language understanding (SLU) that demonstrates robust performance in various zero-shot settings. WHISMA combines the speech encoder from Whisper with the Llama-3 LLM, and is fine-tuned in a parameter-efficient manner on a comprehensive collection of SLU-related datasets. Our experiments show that WHISMA significantly improves the zero-shot slot filling performance on the SLURP benchmark, achieving a relative gain of 26.6% compared to the current state-of-the-art model. Furthermore, to evaluate WHISMA's generalisation capabilities to unseen domains, we develop a new task-agnostic benchmark named SLU-GLUE. The evaluation results indicate that WHISMA outperforms an existing speech-LLM (Qwen-Audio) with a relative gain of 33.0%. |
| title | WHISMA: A Speech-LLM to Perform Zero-shot Spoken Language Understanding |
| topic | Audio and Speech Processing Sound |
| url | https://arxiv.org/abs/2408.16423 |