Low-Resource Cross-Domain Singing Voice Synthesis via Reduced Self-Supervised Speech Representations
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913221384863744 |
|---|---|
| author | Kakoulidis, Panos Ellinas, Nikolaos Vamvoukakis, Georgios Christidou, Myrsini Vioni, Alexandra Maniati, Georgia Oh, Junkwang Jho, Gunu Hwang, Inchul Tsiakoulis, Pirros Chalamandaris, Aimilios |
| author_facet | Kakoulidis, Panos Ellinas, Nikolaos Vamvoukakis, Georgios Christidou, Myrsini Vioni, Alexandra Maniati, Georgia Oh, Junkwang Jho, Gunu Hwang, Inchul Tsiakoulis, Pirros Chalamandaris, Aimilios |
| contents | In this paper, we propose a singing voice synthesis model, Karaoker-SSL, that is trained only on text and speech data as a typical multi-speaker acoustic model. It is a low-resource pipeline that does not utilize any singing data end-to-end, since its vocoder is also trained on speech data. Karaoker-SSL is conditioned by self-supervised speech representations in an unsupervised manner. We preprocess these representations by selecting only a subset of their task-correlated dimensions. The conditioning module is indirectly guided to capture style information during training by multi-tasking. This is achieved with a Conformer-based module, which predicts the pitch from the acoustic model's output. Thus, Karaoker-SSL allows singing voice synthesis without reliance on hand-crafted and domain-specific features. There are also no requirements for text alignments or lyrics timestamps. To refine the voice quality, we employ a U-Net discriminator that is conditioned on the target speaker and follows a Diffusion GAN training scheme. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_01520 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Low-Resource Cross-Domain Singing Voice Synthesis via Reduced Self-Supervised Speech Representations Kakoulidis, Panos Ellinas, Nikolaos Vamvoukakis, Georgios Christidou, Myrsini Vioni, Alexandra Maniati, Georgia Oh, Junkwang Jho, Gunu Hwang, Inchul Tsiakoulis, Pirros Chalamandaris, Aimilios Sound Machine Learning Audio and Speech Processing In this paper, we propose a singing voice synthesis model, Karaoker-SSL, that is trained only on text and speech data as a typical multi-speaker acoustic model. It is a low-resource pipeline that does not utilize any singing data end-to-end, since its vocoder is also trained on speech data. Karaoker-SSL is conditioned by self-supervised speech representations in an unsupervised manner. We preprocess these representations by selecting only a subset of their task-correlated dimensions. The conditioning module is indirectly guided to capture style information during training by multi-tasking. This is achieved with a Conformer-based module, which predicts the pitch from the acoustic model's output. Thus, Karaoker-SSL allows singing voice synthesis without reliance on hand-crafted and domain-specific features. There are also no requirements for text alignments or lyrics timestamps. To refine the voice quality, we employ a U-Net discriminator that is conditioned on the target speaker and follows a Diffusion GAN training scheme. |
| title | Low-Resource Cross-Domain Singing Voice Synthesis via Reduced Self-Supervised Speech Representations |
| topic | Sound Machine Learning Audio and Speech Processing |
| url | https://arxiv.org/abs/2402.01520 |