Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.14785 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909795252961280 |
|---|---|
| author | Seki, Kentaro Okamoto, Yuki Yamaoka, Kouei Saito, Yuki Takamichi, Shinnosuke Saruwatari, Hiroshi |
| author_facet | Seki, Kentaro Okamoto, Yuki Yamaoka, Kouei Saito, Yuki Takamichi, Shinnosuke Saruwatari, Hiroshi |
| contents | Contrastive language--audio pretraining (CLAP) has achieved remarkable success as an audio--text embedding framework, but existing approaches are limited to monaural or single-source conditions and cannot fully capture spatial information. The central challenge in modeling spatial information lies in multi-source conditions, where the correct correspondence between each sound source and its location is required. To tackle this problem, we propose Spatial-CLAP, which introduces a content-aware spatial encoder that enables spatial representations coupled with audio content. We further propose spatial contrastive learning (SCL), a training strategy that explicitly enforces the learning of the correct correspondence and promotes more reliable embeddings under multi-source conditions. Experimental evaluations, including downstream tasks, demonstrate that Spatial-CLAP learns effective embeddings even under multi-source conditions, and confirm the effectiveness of SCL. Moreover, evaluation on unseen three-source mixtures highlights the fundamental distinction between conventional single-source training and our proposed multi-source training paradigm. These findings establish a new paradigm for spatially-aware audio--text embeddings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_14785 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Spatial-CLAP: Learning Spatially-Aware audio--text Embeddings for Multi-Source Conditions Seki, Kentaro Okamoto, Yuki Yamaoka, Kouei Saito, Yuki Takamichi, Shinnosuke Saruwatari, Hiroshi Sound Contrastive language--audio pretraining (CLAP) has achieved remarkable success as an audio--text embedding framework, but existing approaches are limited to monaural or single-source conditions and cannot fully capture spatial information. The central challenge in modeling spatial information lies in multi-source conditions, where the correct correspondence between each sound source and its location is required. To tackle this problem, we propose Spatial-CLAP, which introduces a content-aware spatial encoder that enables spatial representations coupled with audio content. We further propose spatial contrastive learning (SCL), a training strategy that explicitly enforces the learning of the correct correspondence and promotes more reliable embeddings under multi-source conditions. Experimental evaluations, including downstream tasks, demonstrate that Spatial-CLAP learns effective embeddings even under multi-source conditions, and confirm the effectiveness of SCL. Moreover, evaluation on unseen three-source mixtures highlights the fundamental distinction between conventional single-source training and our proposed multi-source training paradigm. These findings establish a new paradigm for spatially-aware audio--text embeddings. |
| title | Spatial-CLAP: Learning Spatially-Aware audio--text Embeddings for Multi-Source Conditions |
| topic | Sound |
| url | https://arxiv.org/abs/2509.14785 |