Versatile Framework for Song Generation with Prompt-based Control
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910022575849472 |
|---|---|
| author | Zhang, Yu Guo, Wenxiang Pan, Changhao Zhu, Zhiyuan Li, Ruiqi Lu, Jingyu Huang, Rongjie Zhang, Ruiyuan Hong, Zhiqing Jiang, Ziyue Zhao, Zhou |
| author_facet | Zhang, Yu Guo, Wenxiang Pan, Changhao Zhu, Zhiyuan Li, Ruiqi Lu, Jingyu Huang, Rongjie Zhang, Ruiyuan Hong, Zhiqing Jiang, Ziyue Zhao, Zhou |
| contents | Song generation focuses on producing controllable high-quality songs based on various prompts. However, existing methods struggle to generate vocals and accompaniments with prompt-based control and proper alignment. Additionally, they fall short in supporting various tasks. To address these challenges, we introduce VersBand, a multi-task song generation framework for synthesizing high-quality, aligned songs with prompt-based control. VersBand comprises these primary models: 1) VocalBand, a decoupled model, leverages the flow-matching method for generating singing styles, pitches, and mel-spectrograms, allowing fast, high-quality vocal generation with style control. 2) AccompBand, a flow-based transformer model, incorporates the Band-MOE, selecting suitable experts for enhanced quality, alignment, and control. This model allows for generating controllable, high-quality accompaniments aligned with vocals. 3) Two generation models, LyricBand for lyrics and MelodyBand for melodies, contribute to the comprehensive multi-task song generation system, allowing for extensive control based on multiple prompts. Experimental results show that VersBand outperforms baseline models across multiple song generation tasks using objective and subjective metrics. Demos and codes are available at https://aaronz345.github.io/VersBandDemo and https://github.com/AaronZ345/VersBand. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_19062 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Versatile Framework for Song Generation with Prompt-based Control Zhang, Yu Guo, Wenxiang Pan, Changhao Zhu, Zhiyuan Li, Ruiqi Lu, Jingyu Huang, Rongjie Zhang, Ruiyuan Hong, Zhiqing Jiang, Ziyue Zhao, Zhou Audio and Speech Processing Computation and Language Sound Song generation focuses on producing controllable high-quality songs based on various prompts. However, existing methods struggle to generate vocals and accompaniments with prompt-based control and proper alignment. Additionally, they fall short in supporting various tasks. To address these challenges, we introduce VersBand, a multi-task song generation framework for synthesizing high-quality, aligned songs with prompt-based control. VersBand comprises these primary models: 1) VocalBand, a decoupled model, leverages the flow-matching method for generating singing styles, pitches, and mel-spectrograms, allowing fast, high-quality vocal generation with style control. 2) AccompBand, a flow-based transformer model, incorporates the Band-MOE, selecting suitable experts for enhanced quality, alignment, and control. This model allows for generating controllable, high-quality accompaniments aligned with vocals. 3) Two generation models, LyricBand for lyrics and MelodyBand for melodies, contribute to the comprehensive multi-task song generation system, allowing for extensive control based on multiple prompts. Experimental results show that VersBand outperforms baseline models across multiple song generation tasks using objective and subjective metrics. Demos and codes are available at https://aaronz345.github.io/VersBandDemo and https://github.com/AaronZ345/VersBand. |
| title | Versatile Framework for Song Generation with Prompt-based Control |
| topic | Audio and Speech Processing Computation and Language Sound |
| url | https://arxiv.org/abs/2504.19062 |