SEAL: Speech Embedding Alignment Learning for Speech Large Language Model with Retrieval-Augmented Generation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911311030386688 |
|---|---|
| author | Sun, Chunyu Liu, Bingyu Cui, Zhichao Shi, Junhan Qi, Anbin Zhang, Tian-hao Zhou, Dinghao Lu, Lewei |
| author_facet | Sun, Chunyu Liu, Bingyu Cui, Zhichao Shi, Junhan Qi, Anbin Zhang, Tian-hao Zhou, Dinghao Lu, Lewei |
| contents | Embedding-based retrieval models have made significant strides in retrieval-augmented generation (RAG) techniques for text and multimodal large language models (LLMs) applications. However, when it comes to speech larage language models (SLLMs), these methods are limited to a two-stage process, where automatic speech recognition (ASR) is combined with text-based retrieval. This sequential architecture suffers from high latency and error propagation. To address these limitations, we propose a unified embedding framework that eliminates the need for intermediate text representations. Specifically, the framework includes separate speech and text encoders, followed by a shared scaling layer that maps both modalities into a common embedding space. Our model reduces pipeline latency by 50\% while achieving higher retrieval accuracy compared to traditional two-stage methods. We also provide a theoretical analysis of the challenges inherent in end-to-end speech retrieval and introduce architectural principles for effective speech-to-document matching. Extensive experiments demonstrate the robustness of our approach across diverse acoustic conditions and speaker variations, paving the way for a new paradigm in multimodal SLLMs retrieval systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_02603 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SEAL: Speech Embedding Alignment Learning for Speech Large Language Model with Retrieval-Augmented Generation Sun, Chunyu Liu, Bingyu Cui, Zhichao Shi, Junhan Qi, Anbin Zhang, Tian-hao Zhou, Dinghao Lu, Lewei Audio and Speech Processing Computation and Language Sound Embedding-based retrieval models have made significant strides in retrieval-augmented generation (RAG) techniques for text and multimodal large language models (LLMs) applications. However, when it comes to speech larage language models (SLLMs), these methods are limited to a two-stage process, where automatic speech recognition (ASR) is combined with text-based retrieval. This sequential architecture suffers from high latency and error propagation. To address these limitations, we propose a unified embedding framework that eliminates the need for intermediate text representations. Specifically, the framework includes separate speech and text encoders, followed by a shared scaling layer that maps both modalities into a common embedding space. Our model reduces pipeline latency by 50\% while achieving higher retrieval accuracy compared to traditional two-stage methods. We also provide a theoretical analysis of the challenges inherent in end-to-end speech retrieval and introduce architectural principles for effective speech-to-document matching. Extensive experiments demonstrate the robustness of our approach across diverse acoustic conditions and speaker variations, paving the way for a new paradigm in multimodal SLLMs retrieval systems. |
| title | SEAL: Speech Embedding Alignment Learning for Speech Large Language Model with Retrieval-Augmented Generation |
| topic | Audio and Speech Processing Computation and Language Sound |
| url | https://arxiv.org/abs/2502.02603 |