Investigating Disentanglement in a Phoneme-level Speech Codec for Prosody Modeling
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910602524360704 |
|---|---|
| author | Karapiperis, Sotirios Ellinas, Nikolaos Vioni, Alexandra Oh, Junkwang Jho, Gunu Hwang, Inchul Raptis, Spyros |
| author_facet | Karapiperis, Sotirios Ellinas, Nikolaos Vioni, Alexandra Oh, Junkwang Jho, Gunu Hwang, Inchul Raptis, Spyros |
| contents | Most of the prevalent approaches in speech prosody modeling rely on learning global style representations in a continuous latent space which encode and transfer the attributes of reference speech. However, recent work on neural codecs which are based on Residual Vector Quantization (RVQ) already shows great potential offering distinct advantages. We investigate the prosody modeling capabilities of the discrete space of such an RVQ-VAE model, modifying it to operate on the phoneme-level. We condition both the encoder and decoder of the model on linguistic representations and apply a global speaker embedding in order to factor out both phonetic and speaker information. We conduct an extensive set of investigations based on subjective experiments and objective measures to show that the phoneme-level discrete latent representations obtained this way achieves a high degree of disentanglement, capturing fine-grained prosodic information that is robust and transferable. The latent space turns out to have interpretable structure with its principal components corresponding to pitch and energy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_08664 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Investigating Disentanglement in a Phoneme-level Speech Codec for Prosody Modeling Karapiperis, Sotirios Ellinas, Nikolaos Vioni, Alexandra Oh, Junkwang Jho, Gunu Hwang, Inchul Raptis, Spyros Sound Computation and Language Machine Learning Audio and Speech Processing Most of the prevalent approaches in speech prosody modeling rely on learning global style representations in a continuous latent space which encode and transfer the attributes of reference speech. However, recent work on neural codecs which are based on Residual Vector Quantization (RVQ) already shows great potential offering distinct advantages. We investigate the prosody modeling capabilities of the discrete space of such an RVQ-VAE model, modifying it to operate on the phoneme-level. We condition both the encoder and decoder of the model on linguistic representations and apply a global speaker embedding in order to factor out both phonetic and speaker information. We conduct an extensive set of investigations based on subjective experiments and objective measures to show that the phoneme-level discrete latent representations obtained this way achieves a high degree of disentanglement, capturing fine-grained prosodic information that is robust and transferable. The latent space turns out to have interpretable structure with its principal components corresponding to pitch and energy. |
| title | Investigating Disentanglement in a Phoneme-level Speech Codec for Prosody Modeling |
| topic | Sound Computation and Language Machine Learning Audio and Speech Processing |
| url | https://arxiv.org/abs/2409.08664 |