DualVC 2: Dynamic Masked Convolution for Unified Streaming and Non-Streaming Voice Conversion
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909076398538752 |
|---|---|
| author | Ning, Ziqian Jiang, Yuepeng Zhu, Pengcheng Wang, Shuai Yao, Jixun Xie, Lei Bi, Mengxiao |
| author_facet | Ning, Ziqian Jiang, Yuepeng Zhu, Pengcheng Wang, Shuai Yao, Jixun Xie, Lei Bi, Mengxiao |
| contents | Voice conversion is becoming increasingly popular, and a growing number of application scenarios require models with streaming inference capabilities. The recently proposed DualVC attempts to achieve this objective through streaming model architecture design and intra-model knowledge distillation along with hybrid predictive coding to compensate for the lack of future information. However, DualVC encounters several problems that limit its performance. First, the autoregressive decoder has error accumulation in its nature and limits the inference speed as well. Second, the causal convolution enables streaming capability but cannot sufficiently use future information within chunks. Third, the model is unable to effectively address the noise in the unvoiced segments, lowering the sound quality. In this paper, we propose DualVC 2 to address these issues. Specifically, the model backbone is migrated to a Conformer-based architecture, empowering parallel inference. Causal convolution is replaced by non-causal convolution with dynamic chunk mask to make better use of within-chunk future information. Also, quiet attention is introduced to enhance the model's noise robustness. Experiments show that DualVC 2 outperforms DualVC and other baseline systems in both subjective and objective metrics, with only 186.4 ms latency. Our audio samples are made publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_15496 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | DualVC 2: Dynamic Masked Convolution for Unified Streaming and Non-Streaming Voice Conversion Ning, Ziqian Jiang, Yuepeng Zhu, Pengcheng Wang, Shuai Yao, Jixun Xie, Lei Bi, Mengxiao Audio and Speech Processing Sound Voice conversion is becoming increasingly popular, and a growing number of application scenarios require models with streaming inference capabilities. The recently proposed DualVC attempts to achieve this objective through streaming model architecture design and intra-model knowledge distillation along with hybrid predictive coding to compensate for the lack of future information. However, DualVC encounters several problems that limit its performance. First, the autoregressive decoder has error accumulation in its nature and limits the inference speed as well. Second, the causal convolution enables streaming capability but cannot sufficiently use future information within chunks. Third, the model is unable to effectively address the noise in the unvoiced segments, lowering the sound quality. In this paper, we propose DualVC 2 to address these issues. Specifically, the model backbone is migrated to a Conformer-based architecture, empowering parallel inference. Causal convolution is replaced by non-causal convolution with dynamic chunk mask to make better use of within-chunk future information. Also, quiet attention is introduced to enhance the model's noise robustness. Experiments show that DualVC 2 outperforms DualVC and other baseline systems in both subjective and objective metrics, with only 186.4 ms latency. Our audio samples are made publicly available. |
| title | DualVC 2: Dynamic Masked Convolution for Unified Streaming and Non-Streaming Voice Conversion |
| topic | Audio and Speech Processing Sound |
| url | https://arxiv.org/abs/2309.15496 |