Can Large Language Models Understand Spatial Audio?
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917693938991104 |
|---|---|
| author | Tang, Changli Yu, Wenyi Sun, Guangzhi Chen, Xianzhao Tan, Tian Li, Wei Zhang, Jun Lu, Lu Ma, Zejun Wang, Yuxuan Zhang, Chao |
| author_facet | Tang, Changli Yu, Wenyi Sun, Guangzhi Chen, Xianzhao Tan, Tian Li, Wei Zhang, Jun Lu, Lu Ma, Zejun Wang, Yuxuan Zhang, Chao |
| contents | This paper explores enabling large language models (LLMs) to understand spatial information from multichannel audio, a skill currently lacking in auditory LLMs. By leveraging LLMs' advanced cognitive and inferential abilities, the aim is to enhance understanding of 3D environments via audio. We study 3 spatial audio tasks: sound source localization (SSL), far-field speech recognition (FSR), and localisation-informed speech extraction (LSE), achieving notable progress in each task. For SSL, our approach achieves an MAE of $2.70^{\circ}$ on the Spatial LibriSpeech dataset, substantially surpassing the prior benchmark of about $6.60^{\circ}$. Moreover, our model can employ spatial cues to improve FSR accuracy and execute LSE by selectively attending to sounds originating from a specified direction via text prompts, even amidst overlapping speech. These findings highlight the potential of adapting LLMs to grasp physical audio concepts, paving the way for LLM-based agents in 3D environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_07914 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Can Large Language Models Understand Spatial Audio? Tang, Changli Yu, Wenyi Sun, Guangzhi Chen, Xianzhao Tan, Tian Li, Wei Zhang, Jun Lu, Lu Ma, Zejun Wang, Yuxuan Zhang, Chao Sound Audio and Speech Processing This paper explores enabling large language models (LLMs) to understand spatial information from multichannel audio, a skill currently lacking in auditory LLMs. By leveraging LLMs' advanced cognitive and inferential abilities, the aim is to enhance understanding of 3D environments via audio. We study 3 spatial audio tasks: sound source localization (SSL), far-field speech recognition (FSR), and localisation-informed speech extraction (LSE), achieving notable progress in each task. For SSL, our approach achieves an MAE of $2.70^{\circ}$ on the Spatial LibriSpeech dataset, substantially surpassing the prior benchmark of about $6.60^{\circ}$. Moreover, our model can employ spatial cues to improve FSR accuracy and execute LSE by selectively attending to sounds originating from a specified direction via text prompts, even amidst overlapping speech. These findings highlight the potential of adapting LLMs to grasp physical audio concepts, paving the way for LLM-based agents in 3D environments. |
| title | Can Large Language Models Understand Spatial Audio? |
| topic | Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2406.07914 |