Thinking in Directivity: Speech Large Language Model for Multi-Talker Directional Speech Recognition
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916797706403840 |
|---|---|
| author | Xie, Jiamin Lin, Ju Huang, Yiteng Vuong, Tyler Lin, Zhaojiang Yang, Zhaojun Su, Peng Rawat, Prashant Srivastava, Sangeeta Sun, Ming Metze, Florian |
| author_facet | Xie, Jiamin Lin, Ju Huang, Yiteng Vuong, Tyler Lin, Zhaojiang Yang, Zhaojun Su, Peng Rawat, Prashant Srivastava, Sangeeta Sun, Ming Metze, Florian |
| contents | Recent studies have demonstrated that prompting large language models (LLM) with audio encodings enables effective speech recognition capabilities. However, the ability of Speech LLMs to comprehend and process multi-channel audio with spatial cues remains a relatively uninvestigated area of research. In this work, we present directional-SpeechLlama, a novel approach that leverages the microphone array of smart glasses to achieve directional speech recognition, source localization, and bystander cross-talk suppression. To enhance the model's ability to understand directivity, we propose two key techniques: serialized directional output training (S-DOT) and contrastive direction data augmentation (CDDA). Experimental results show that our proposed directional-SpeechLlama effectively captures the relationship between textual cues and spatial audio, yielding strong performance in both speech recognition and source localization tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_14973 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Thinking in Directivity: Speech Large Language Model for Multi-Talker Directional Speech Recognition Xie, Jiamin Lin, Ju Huang, Yiteng Vuong, Tyler Lin, Zhaojiang Yang, Zhaojun Su, Peng Rawat, Prashant Srivastava, Sangeeta Sun, Ming Metze, Florian Audio and Speech Processing Artificial Intelligence Recent studies have demonstrated that prompting large language models (LLM) with audio encodings enables effective speech recognition capabilities. However, the ability of Speech LLMs to comprehend and process multi-channel audio with spatial cues remains a relatively uninvestigated area of research. In this work, we present directional-SpeechLlama, a novel approach that leverages the microphone array of smart glasses to achieve directional speech recognition, source localization, and bystander cross-talk suppression. To enhance the model's ability to understand directivity, we propose two key techniques: serialized directional output training (S-DOT) and contrastive direction data augmentation (CDDA). Experimental results show that our proposed directional-SpeechLlama effectively captures the relationship between textual cues and spatial audio, yielding strong performance in both speech recognition and source localization tasks. |
| title | Thinking in Directivity: Speech Large Language Model for Multi-Talker Directional Speech Recognition |
| topic | Audio and Speech Processing Artificial Intelligence |
| url | https://arxiv.org/abs/2506.14973 |