SELM: Speech Enhancement Using Discrete Tokens and Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911749959057408 |
|---|---|
| author | Wang, Ziqian Zhu, Xinfa Zhang, Zihan Lv, YuanJun Jiang, Ning Zhao, Guoqing Xie, Lei |
| author_facet | Wang, Ziqian Zhu, Xinfa Zhang, Zihan Lv, YuanJun Jiang, Ning Zhao, Guoqing Xie, Lei |
| contents | Language models (LMs) have shown superior performances in various speech generation tasks recently, demonstrating their powerful ability for semantic context modeling. Given the intrinsic similarity between speech generation and speech enhancement, harnessing semantic information holds potential advantages for speech enhancement tasks. In light of this, we propose SELM, a novel paradigm for speech enhancement, which integrates discrete tokens and leverages language models. SELM comprises three stages: encoding, modeling, and decoding. We transform continuous waveform signals into discrete tokens using pre-trained self-supervised learning (SSL) models and a k-means tokenizer. Language models then capture comprehensive contextual information within these tokens. Finally, a detokenizer and HiFi-GAN restore them into enhanced speech. Experimental results demonstrate that SELM achieves comparable performance in objective metrics alongside superior results in subjective perception. Our demos are available https://honee-w.github.io/SELM/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_09747 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | SELM: Speech Enhancement Using Discrete Tokens and Language Models Wang, Ziqian Zhu, Xinfa Zhang, Zihan Lv, YuanJun Jiang, Ning Zhao, Guoqing Xie, Lei Audio and Speech Processing Signal Processing Language models (LMs) have shown superior performances in various speech generation tasks recently, demonstrating their powerful ability for semantic context modeling. Given the intrinsic similarity between speech generation and speech enhancement, harnessing semantic information holds potential advantages for speech enhancement tasks. In light of this, we propose SELM, a novel paradigm for speech enhancement, which integrates discrete tokens and leverages language models. SELM comprises three stages: encoding, modeling, and decoding. We transform continuous waveform signals into discrete tokens using pre-trained self-supervised learning (SSL) models and a k-means tokenizer. Language models then capture comprehensive contextual information within these tokens. Finally, a detokenizer and HiFi-GAN restore them into enhanced speech. Experimental results demonstrate that SELM achieves comparable performance in objective metrics alongside superior results in subjective perception. Our demos are available https://honee-w.github.io/SELM/. |
| title | SELM: Speech Enhancement Using Discrete Tokens and Language Models |
| topic | Audio and Speech Processing Signal Processing |
| url | https://arxiv.org/abs/2312.09747 |