UniSep: Universal Target Audio Separation with Language Models at Scale
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916667774205952 |
|---|---|
| author | Wang, Yuanyuan Chen, Hangting Yang, Dongchao Li, Weiqin Luo, Dan Li, Guangzhi Yang, Shan Wu, Zhiyong Meng, Helen Wu, Xixin |
| author_facet | Wang, Yuanyuan Chen, Hangting Yang, Dongchao Li, Weiqin Luo, Dan Li, Guangzhi Yang, Shan Wu, Zhiyong Meng, Helen Wu, Xixin |
| contents | We propose Universal target audio Separation (UniSep), addressing the separation task on arbitrary mixtures of different types of audio. Distinguished from previous studies, UniSep is performed on unlimited source domains and unlimited source numbers. We formulate the separation task as a sequence-to-sequence problem, and a large language model (LLM) is used to model the audio sequence in the discrete latent space, leveraging the power of LLM in handling complex mixture audios with large-scale data. Moreover, a novel pre-training strategy is proposed to utilize audio-only data, which reduces the efforts of large-scale data simulation and enhances the ability of LLMs to understand the consistency and correlation of information within audio sequences. We also demonstrate the effectiveness of scaling datasets in an audio separation task: we use large-scale data (36.5k hours), including speech, music, and sound, to train a universal target audio separation model that is not limited to a specific domain. Experiments show that UniSep achieves competitive subjective and objective evaluation results compared with single-task models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_23762 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | UniSep: Universal Target Audio Separation with Language Models at Scale Wang, Yuanyuan Chen, Hangting Yang, Dongchao Li, Weiqin Luo, Dan Li, Guangzhi Yang, Shan Wu, Zhiyong Meng, Helen Wu, Xixin Sound Audio and Speech Processing We propose Universal target audio Separation (UniSep), addressing the separation task on arbitrary mixtures of different types of audio. Distinguished from previous studies, UniSep is performed on unlimited source domains and unlimited source numbers. We formulate the separation task as a sequence-to-sequence problem, and a large language model (LLM) is used to model the audio sequence in the discrete latent space, leveraging the power of LLM in handling complex mixture audios with large-scale data. Moreover, a novel pre-training strategy is proposed to utilize audio-only data, which reduces the efforts of large-scale data simulation and enhances the ability of LLMs to understand the consistency and correlation of information within audio sequences. We also demonstrate the effectiveness of scaling datasets in an audio separation task: we use large-scale data (36.5k hours), including speech, music, and sound, to train a universal target audio separation model that is not limited to a specific domain. Experiments show that UniSep achieves competitive subjective and objective evaluation results compared with single-task models. |
| title | UniSep: Universal Target Audio Separation with Language Models at Scale |
| topic | Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2503.23762 |