Optimal Transport Maps are Good Voice Converters
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929576319385600 |
|---|---|
| author | Asadulaev, Arip Korst, Rostislav Shutov, Vitalii Korotin, Alexander Grebnyak, Yaroslav Egiazarian, Vahe Burnaev, Evgeny |
| author_facet | Asadulaev, Arip Korst, Rostislav Shutov, Vitalii Korotin, Alexander Grebnyak, Yaroslav Egiazarian, Vahe Burnaev, Evgeny |
| contents | Recently, neural network-based methods for computing optimal transport maps have been effectively applied to style transfer problems. However, the application of these methods to voice conversion is underexplored. In our paper, we fill this gap by investigating optimal transport as a framework for voice conversion. We present a variety of optimal transport algorithms designed for different data representations, such as mel-spectrograms and latent representation of self-supervised speech models. For the mel-spectogram data representation, we achieve strong results in terms of Frechet Audio Distance (FAD). This performance is consistent with our theoretical analysis, which suggests that our method provides an upper bound on the FAD between the target and generated distributions. Within the latent space of the WavLM encoder, we achived state-of-the-art results and outperformed existing methods even with limited reference speaker data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_02402 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Optimal Transport Maps are Good Voice Converters Asadulaev, Arip Korst, Rostislav Shutov, Vitalii Korotin, Alexander Grebnyak, Yaroslav Egiazarian, Vahe Burnaev, Evgeny Sound Machine Learning Audio and Speech Processing Recently, neural network-based methods for computing optimal transport maps have been effectively applied to style transfer problems. However, the application of these methods to voice conversion is underexplored. In our paper, we fill this gap by investigating optimal transport as a framework for voice conversion. We present a variety of optimal transport algorithms designed for different data representations, such as mel-spectrograms and latent representation of self-supervised speech models. For the mel-spectogram data representation, we achieve strong results in terms of Frechet Audio Distance (FAD). This performance is consistent with our theoretical analysis, which suggests that our method provides an upper bound on the FAD between the target and generated distributions. Within the latent space of the WavLM encoder, we achived state-of-the-art results and outperformed existing methods even with limited reference speaker data. |
| title | Optimal Transport Maps are Good Voice Converters |
| topic | Sound Machine Learning Audio and Speech Processing |
| url | https://arxiv.org/abs/2411.02402 |