TSELM: Target Speaker Extraction using Discrete Tokens and Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929501205692416 |
|---|---|
| author | Tang, Beilong Zeng, Bang Li, Ming |
| author_facet | Tang, Beilong Zeng, Bang Li, Ming |
| contents | We propose TSELM, a novel target speaker extraction network that leverages discrete tokens and language models. TSELM utilizes multiple discretized layers from WavLM as input tokens and incorporates cross-attention mechanisms to integrate target speaker information. Language models are employed to capture the sequence dependencies, while a scalable HiFi-GAN is used to reconstruct the audio from the tokens. By applying a cross-entropy loss, TSELM models the probability distribution of output tokens, thus converting the complex regression problem of audio generation into a classification task. Experimental results show that TSELM achieves excellent results in speech quality and comparable results in speech intelligibility. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_07841 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TSELM: Target Speaker Extraction using Discrete Tokens and Language Models Tang, Beilong Zeng, Bang Li, Ming Sound Machine Learning Audio and Speech Processing We propose TSELM, a novel target speaker extraction network that leverages discrete tokens and language models. TSELM utilizes multiple discretized layers from WavLM as input tokens and incorporates cross-attention mechanisms to integrate target speaker information. Language models are employed to capture the sequence dependencies, while a scalable HiFi-GAN is used to reconstruct the audio from the tokens. By applying a cross-entropy loss, TSELM models the probability distribution of output tokens, thus converting the complex regression problem of audio generation into a classification task. Experimental results show that TSELM achieves excellent results in speech quality and comparable results in speech intelligibility. |
| title | TSELM: Target Speaker Extraction using Discrete Tokens and Language Models |
| topic | Sound Machine Learning Audio and Speech Processing |
| url | https://arxiv.org/abs/2409.07841 |