Enhancing Expressive Voice Conversion with Discrete Pitch-Conditioned Flow Matching Model
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916604539830272 |
|---|---|
| author | Zuo, Jialong Ji, Shengpeng Fang, Minghui Jiang, Ziyue Cheng, Xize Yang, Qian Liu, Wenrui Zhang, Guangyan Tu, Zehai Guo, Yiwen Zhao, Zhou |
| author_facet | Zuo, Jialong Ji, Shengpeng Fang, Minghui Jiang, Ziyue Cheng, Xize Yang, Qian Liu, Wenrui Zhang, Guangyan Tu, Zehai Guo, Yiwen Zhao, Zhou |
| contents | This paper introduces PFlow-VC, a conditional flow matching voice conversion model that leverages fine-grained discrete pitch tokens and target speaker prompt information for expressive voice conversion (VC). Previous VC works primarily focus on speaker conversion, with further exploration needed in enhancing expressiveness (such as prosody and emotion) for timbre conversion. Unlike previous methods, we adopt a simple and efficient approach to enhance the style expressiveness of voice conversion models. Specifically, we pretrain a self-supervised pitch VQVAE model to discretize speaker-irrelevant pitch information and leverage a masked pitch-conditioned flow matching model for Mel-spectrogram synthesis, which provides in-context pitch modeling capabilities for the speaker conversion model, effectively improving the voice style transfer capacity. Additionally, we improve timbre similarity by combining global timbre embeddings with time-varying timbre tokens. Experiments on unseen LibriTTS test-clean and emotional speech dataset ESD show the superiority of the PFlow-VC model in both timbre conversion and style transfer. Audio samples are available on the demo page https://speechai-demo.github.io/PFlow-VC/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_05471 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Enhancing Expressive Voice Conversion with Discrete Pitch-Conditioned Flow Matching Model Zuo, Jialong Ji, Shengpeng Fang, Minghui Jiang, Ziyue Cheng, Xize Yang, Qian Liu, Wenrui Zhang, Guangyan Tu, Zehai Guo, Yiwen Zhao, Zhou Sound Audio and Speech Processing This paper introduces PFlow-VC, a conditional flow matching voice conversion model that leverages fine-grained discrete pitch tokens and target speaker prompt information for expressive voice conversion (VC). Previous VC works primarily focus on speaker conversion, with further exploration needed in enhancing expressiveness (such as prosody and emotion) for timbre conversion. Unlike previous methods, we adopt a simple and efficient approach to enhance the style expressiveness of voice conversion models. Specifically, we pretrain a self-supervised pitch VQVAE model to discretize speaker-irrelevant pitch information and leverage a masked pitch-conditioned flow matching model for Mel-spectrogram synthesis, which provides in-context pitch modeling capabilities for the speaker conversion model, effectively improving the voice style transfer capacity. Additionally, we improve timbre similarity by combining global timbre embeddings with time-varying timbre tokens. Experiments on unseen LibriTTS test-clean and emotional speech dataset ESD show the superiority of the PFlow-VC model in both timbre conversion and style transfer. Audio samples are available on the demo page https://speechai-demo.github.io/PFlow-VC/. |
| title | Enhancing Expressive Voice Conversion with Discrete Pitch-Conditioned Flow Matching Model |
| topic | Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2502.05471 |