Diffusion-Based Speech Enhancement with Joint Generative and Predictive Decoders
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916141451968512 |
|---|---|
| author | Shi, Hao Shimada, Kazuki Hirano, Masato Shibuya, Takashi Koyama, Yuichiro Zhong, Zhi Takahashi, Shusuke Kawahara, Tatsuya Mitsufuji, Yuki |
| author_facet | Shi, Hao Shimada, Kazuki Hirano, Masato Shibuya, Takashi Koyama, Yuichiro Zhong, Zhi Takahashi, Shusuke Kawahara, Tatsuya Mitsufuji, Yuki |
| contents | Diffusion-based generative speech enhancement (SE) has recently received attention, but reverse diffusion remains time-consuming. One solution is to initialize the reverse diffusion process with enhanced features estimated by a predictive SE system. However, the pipeline structure currently does not consider for a combined use of generative and predictive decoders. The predictive decoder allows us to use the further complementarity between predictive and diffusion-based generative SE. In this paper, we propose a unified system that use jointly generative and predictive decoders across two levels. The encoder encodes both generative and predictive information at the shared encoding level. At the decoded feature level, we fuse the two decoded features by generative and predictive decoders. Specifically, the two SE modules are fused in the initial and final diffusion steps: the initial fusion initializes the diffusion process with the predictive SE to improve convergence, and the final fusion combines the two complementary SE outputs to enhance SE performance. Experiments conducted on the Voice-Bank dataset demonstrate that incorporating predictive information leads to faster decoding and higher PESQ scores compared with other score-based diffusion SE (StoRM and SGMSE+). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_10734 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Diffusion-Based Speech Enhancement with Joint Generative and Predictive Decoders Shi, Hao Shimada, Kazuki Hirano, Masato Shibuya, Takashi Koyama, Yuichiro Zhong, Zhi Takahashi, Shusuke Kawahara, Tatsuya Mitsufuji, Yuki Sound Computation and Language Audio and Speech Processing Diffusion-based generative speech enhancement (SE) has recently received attention, but reverse diffusion remains time-consuming. One solution is to initialize the reverse diffusion process with enhanced features estimated by a predictive SE system. However, the pipeline structure currently does not consider for a combined use of generative and predictive decoders. The predictive decoder allows us to use the further complementarity between predictive and diffusion-based generative SE. In this paper, we propose a unified system that use jointly generative and predictive decoders across two levels. The encoder encodes both generative and predictive information at the shared encoding level. At the decoded feature level, we fuse the two decoded features by generative and predictive decoders. Specifically, the two SE modules are fused in the initial and final diffusion steps: the initial fusion initializes the diffusion process with the predictive SE to improve convergence, and the final fusion combines the two complementary SE outputs to enhance SE performance. Experiments conducted on the Voice-Bank dataset demonstrate that incorporating predictive information leads to faster decoding and higher PESQ scores compared with other score-based diffusion SE (StoRM and SGMSE+). |
| title | Diffusion-Based Speech Enhancement with Joint Generative and Predictive Decoders |
| topic | Sound Computation and Language Audio and Speech Processing |
| url | https://arxiv.org/abs/2305.10734 |