Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.16724 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918143582011392 |
|---|---|
| author | Hu, Jinbo Cao, Yin Wu, Ming Luo, Zhenbo Yang, Jun |
| author_facet | Hu, Jinbo Cao, Yin Wu, Ming Luo, Zhenbo Yang, Jun |
| contents | Spatial audio understanding is essential for accurately perceiving and interpreting acoustic environments. However, existing audio-language models exhibit limitations in processing spatial audio and perceiving spatial acoustic scenes. To address this gap, we propose the Spatial Audio Language Model (SALM), a novel framework that bridges spatial audio and language through multi-modal contrastive learning. SALM integrates a text encoder with a dual-branch audio encoder that decomposes spatial sound into semantic and spatial components via structured audio embeddings. Key features of SALM include seamless alignment between spatial audio and natural language, both separate and joint extraction of spatial and semantic representations, zero-shot direction classification, and flexible support for spatial audio editing. Experimental results demonstrate that SALM effectively captures and aligns cross-modal representations, yielding well-structured audio embeddings. Furthermore, SALM enables advanced editing capabilities, such as modifying directional audio using text-based embeddings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_16724 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SALM: Spatial Audio Language Model with Structured Embeddings for Understanding and Editing Hu, Jinbo Cao, Yin Wu, Ming Luo, Zhenbo Yang, Jun Sound Audio and Speech Processing Spatial audio understanding is essential for accurately perceiving and interpreting acoustic environments. However, existing audio-language models exhibit limitations in processing spatial audio and perceiving spatial acoustic scenes. To address this gap, we propose the Spatial Audio Language Model (SALM), a novel framework that bridges spatial audio and language through multi-modal contrastive learning. SALM integrates a text encoder with a dual-branch audio encoder that decomposes spatial sound into semantic and spatial components via structured audio embeddings. Key features of SALM include seamless alignment between spatial audio and natural language, both separate and joint extraction of spatial and semantic representations, zero-shot direction classification, and flexible support for spatial audio editing. Experimental results demonstrate that SALM effectively captures and aligns cross-modal representations, yielding well-structured audio embeddings. Furthermore, SALM enables advanced editing capabilities, such as modifying directional audio using text-based embeddings. |
| title | SALM: Spatial Audio Language Model with Structured Embeddings for Understanding and Editing |
| topic | Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2507.16724 |