Equipping LLM with Directional Multi-Talker Speech Understanding Capabilities
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917255064846336 |
|---|---|
| author | Lin, Ju Pan, Jing Li, Ruizhi Sun, Ming Liu, Yuzong Hassan, Alaa Zheng, Jing Metze, Florian |
| author_facet | Lin, Ju Pan, Jing Li, Ruizhi Sun, Ming Liu, Yuzong Hassan, Alaa Zheng, Jing Metze, Florian |
| contents | Recent studies have demonstrated that prompting large language models (LLM) with audio encodings enables effective speech understanding capabilities. However, most speech LLMs are trained on single-channel, single-talker data, which makes it challenging to directly apply them to multi-talker and multi-channel speech understanding task. In this work, we present a comprehensive investigation on how to enable directional multi-talker speech understanding capabilities for LLMs, specifically in smart glasses usecase. We propose two novel approaches to integrate directivity into LLMs: (1) a cascaded system that leverages a source separation front-end module, and (2) an end-to-end system that utilizes serialized output training. All of the approaches utilize a multi-microphone array embedded in smart glasses to optimize directivity interpretation and processing in a streaming manner. Experimental results demonstrate the efficacy of our proposed methods in endowing LLMs with directional speech understanding capabilities, achieving strong performance in both speech recognition and speech translation tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_07211 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Equipping LLM with Directional Multi-Talker Speech Understanding Capabilities Lin, Ju Pan, Jing Li, Ruizhi Sun, Ming Liu, Yuzong Hassan, Alaa Zheng, Jing Metze, Florian Computation and Language Sound Recent studies have demonstrated that prompting large language models (LLM) with audio encodings enables effective speech understanding capabilities. However, most speech LLMs are trained on single-channel, single-talker data, which makes it challenging to directly apply them to multi-talker and multi-channel speech understanding task. In this work, we present a comprehensive investigation on how to enable directional multi-talker speech understanding capabilities for LLMs, specifically in smart glasses usecase. We propose two novel approaches to integrate directivity into LLMs: (1) a cascaded system that leverages a source separation front-end module, and (2) an end-to-end system that utilizes serialized output training. All of the approaches utilize a multi-microphone array embedded in smart glasses to optimize directivity interpretation and processing in a streaming manner. Experimental results demonstrate the efficacy of our proposed methods in endowing LLMs with directional speech understanding capabilities, achieving strong performance in both speech recognition and speech translation tasks. |
| title | Equipping LLM with Directional Multi-Talker Speech Understanding Capabilities |
| topic | Computation and Language Sound |
| url | https://arxiv.org/abs/2602.07211 |