Causal Speech Enhancement with Predicting Semantics based on Quantized Self-supervised Learning Features
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909441557790720 |
|---|---|
| author | Tsunoo, Emiru Saito, Yuki Nakata, Wataru Saruwatari, Hiroshi |
| author_facet | Tsunoo, Emiru Saito, Yuki Nakata, Wataru Saruwatari, Hiroshi |
| contents | Real-time speech enhancement (SE) is essential to online speech communication. Causal SE models use only the previous context while predicting future information, such as phoneme continuation, may help performing causal SE. The phonetic information is often represented by quantizing latent features of self-supervised learning (SSL) models. This work is the first to incorporate SSL features with causality into an SE model. The causal SSL features are encoded and combined with spectrogram features using feature-wise linear modulation to estimate a mask for enhancing the noisy input speech. Simultaneously, we quantize the causal SSL features using vector quantization to represent phonetic characteristics as semantic tokens. The model not only encodes SSL features but also predicts the future semantic tokens in multi-task learning (MTL). The experimental results using VoiceBank + DEMAND dataset show that our proposed method achieves 2.88 in PESQ, especially with semantic prediction MTL, in which we confirm that the semantic prediction played an important role in causal SE. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_19248 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Causal Speech Enhancement with Predicting Semantics based on Quantized Self-supervised Learning Features Tsunoo, Emiru Saito, Yuki Nakata, Wataru Saruwatari, Hiroshi Audio and Speech Processing Sound Real-time speech enhancement (SE) is essential to online speech communication. Causal SE models use only the previous context while predicting future information, such as phoneme continuation, may help performing causal SE. The phonetic information is often represented by quantizing latent features of self-supervised learning (SSL) models. This work is the first to incorporate SSL features with causality into an SE model. The causal SSL features are encoded and combined with spectrogram features using feature-wise linear modulation to estimate a mask for enhancing the noisy input speech. Simultaneously, we quantize the causal SSL features using vector quantization to represent phonetic characteristics as semantic tokens. The model not only encodes SSL features but also predicts the future semantic tokens in multi-task learning (MTL). The experimental results using VoiceBank + DEMAND dataset show that our proposed method achieves 2.88 in PESQ, especially with semantic prediction MTL, in which we confirm that the semantic prediction played an important role in causal SE. |
| title | Causal Speech Enhancement with Predicting Semantics based on Quantized Self-supervised Learning Features |
| topic | Audio and Speech Processing Sound |
| url | https://arxiv.org/abs/2412.19248 |