SingOMD: Singing Oriented Multi-resolution Discrete Representation Construction from Speech Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917699313991680 |
|---|---|
| author | Tang, Yuxun Wu, Yuning Shi, Jiatong Jin, Qin |
| author_facet | Tang, Yuxun Wu, Yuning Shi, Jiatong Jin, Qin |
| contents | Discrete representation has shown advantages in speech generation tasks, wherein discrete tokens are derived by discretizing hidden features from self-supervised learning (SSL) pre-trained models. However, the direct application of speech SSL models to singing generation encounters domain gaps between speech and singing. Furthermore, singing generation necessitates a more refined representation than typical speech. To address these challenges, we introduce SingOMD, a novel method to extract singing-oriented multi-resolution discrete representations from speech SSL models. Specifically, we first adapt the features from speech SSL through a resynthesis task and incorporate multi-resolution modules based on resampling to better serve singing generation. These adapted multi-resolution features are then discretized via clustering. Extensive experiments demonstrate the robustness, efficiency, and effectiveness of these representations in singing vocoders and singing voice synthesis. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_08905 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SingOMD: Singing Oriented Multi-resolution Discrete Representation Construction from Speech Models Tang, Yuxun Wu, Yuning Shi, Jiatong Jin, Qin Sound Audio and Speech Processing Discrete representation has shown advantages in speech generation tasks, wherein discrete tokens are derived by discretizing hidden features from self-supervised learning (SSL) pre-trained models. However, the direct application of speech SSL models to singing generation encounters domain gaps between speech and singing. Furthermore, singing generation necessitates a more refined representation than typical speech. To address these challenges, we introduce SingOMD, a novel method to extract singing-oriented multi-resolution discrete representations from speech SSL models. Specifically, we first adapt the features from speech SSL through a resynthesis task and incorporate multi-resolution modules based on resampling to better serve singing generation. These adapted multi-resolution features are then discretized via clustering. Extensive experiments demonstrate the robustness, efficiency, and effectiveness of these representations in singing vocoders and singing voice synthesis. |
| title | SingOMD: Singing Oriented Multi-resolution Discrete Representation Construction from Speech Models |
| topic | Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2406.08905 |