SoloAudio: Target Sound Extraction with Language-oriented Audio Diffusion Transformer
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909445242486784 |
|---|---|
| author | Wang, Helin Hai, Jiarui Lu, Yen-Ju Thakkar, Karan Elhilali, Mounya Dehak, Najim |
| author_facet | Wang, Helin Hai, Jiarui Lu, Yen-Ju Thakkar, Karan Elhilali, Mounya Dehak, Najim |
| contents | In this paper, we introduce SoloAudio, a novel diffusion-based generative model for target sound extraction (TSE). Our approach trains latent diffusion models on audio, replacing the previous U-Net backbone with a skip-connected Transformer that operates on latent features. SoloAudio supports both audio-oriented and language-oriented TSE by utilizing a CLAP model as the feature extractor for target sounds. Furthermore, SoloAudio leverages synthetic audio generated by state-of-the-art text-to-audio models for training, demonstrating strong generalization to out-of-domain data and unseen sound events. We evaluate this approach on the FSD Kaggle 2018 mixture dataset and real data from AudioSet, where SoloAudio achieves the state-of-the-art results on both in-domain and out-of-domain data, and exhibits impressive zero-shot and few-shot capabilities. Source code and demos are released. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_08425 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SoloAudio: Target Sound Extraction with Language-oriented Audio Diffusion Transformer Wang, Helin Hai, Jiarui Lu, Yen-Ju Thakkar, Karan Elhilali, Mounya Dehak, Najim Audio and Speech Processing Sound In this paper, we introduce SoloAudio, a novel diffusion-based generative model for target sound extraction (TSE). Our approach trains latent diffusion models on audio, replacing the previous U-Net backbone with a skip-connected Transformer that operates on latent features. SoloAudio supports both audio-oriented and language-oriented TSE by utilizing a CLAP model as the feature extractor for target sounds. Furthermore, SoloAudio leverages synthetic audio generated by state-of-the-art text-to-audio models for training, demonstrating strong generalization to out-of-domain data and unseen sound events. We evaluate this approach on the FSD Kaggle 2018 mixture dataset and real data from AudioSet, where SoloAudio achieves the state-of-the-art results on both in-domain and out-of-domain data, and exhibits impressive zero-shot and few-shot capabilities. Source code and demos are released. |
| title | SoloAudio: Target Sound Extraction with Language-oriented Audio Diffusion Transformer |
| topic | Audio and Speech Processing Sound |
| url | https://arxiv.org/abs/2409.08425 |