Drop the beat! Freestyler for Accompaniment Conditioned Rapping Voice Generation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916373201944576 |
|---|---|
| author | Ning, Ziqian Wang, Shuai Jiang, Yuepeng Yao, Jixun He, Lei Pan, Shifeng Ding, Jie Xie, Lei |
| author_facet | Ning, Ziqian Wang, Shuai Jiang, Yuepeng Yao, Jixun He, Lei Pan, Shifeng Ding, Jie Xie, Lei |
| contents | Rap, a prominent genre of vocal performance, remains underexplored in vocal generation. General vocal synthesis depends on precise note and duration inputs, requiring users to have related musical knowledge, which limits flexibility. In contrast, rap typically features simpler melodies, with a core focus on a strong rhythmic sense that harmonizes with accompanying beats. In this paper, we propose Freestyler, the first system that generates rapping vocals directly from lyrics and accompaniment inputs. Freestyler utilizes language model-based token generation, followed by a conditional flow matching model to produce spectrograms and a neural vocoder to restore audio. It allows a 3-second prompt to enable zero-shot timbre control. Due to the scarcity of publicly available rap datasets, we also present RapBank, a rap song dataset collected from the internet, alongside a meticulously designed processing pipeline. Experimental results show that Freestyler produces high-quality rapping voice generation with enhanced naturalness and strong alignment with accompanying beats, both stylistically and rhythmically. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_15474 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Drop the beat! Freestyler for Accompaniment Conditioned Rapping Voice Generation Ning, Ziqian Wang, Shuai Jiang, Yuepeng Yao, Jixun He, Lei Pan, Shifeng Ding, Jie Xie, Lei Audio and Speech Processing Sound Rap, a prominent genre of vocal performance, remains underexplored in vocal generation. General vocal synthesis depends on precise note and duration inputs, requiring users to have related musical knowledge, which limits flexibility. In contrast, rap typically features simpler melodies, with a core focus on a strong rhythmic sense that harmonizes with accompanying beats. In this paper, we propose Freestyler, the first system that generates rapping vocals directly from lyrics and accompaniment inputs. Freestyler utilizes language model-based token generation, followed by a conditional flow matching model to produce spectrograms and a neural vocoder to restore audio. It allows a 3-second prompt to enable zero-shot timbre control. Due to the scarcity of publicly available rap datasets, we also present RapBank, a rap song dataset collected from the internet, alongside a meticulously designed processing pipeline. Experimental results show that Freestyler produces high-quality rapping voice generation with enhanced naturalness and strong alignment with accompanying beats, both stylistically and rhythmically. |
| title | Drop the beat! Freestyler for Accompaniment Conditioned Rapping Voice Generation |
| topic | Audio and Speech Processing Sound |
| url | https://arxiv.org/abs/2408.15474 |