MusicFlow: Cascaded Flow Matching for Text Guided Music Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913565555818496 |
|---|---|
| author | Prajwal, K R Shi, Bowen Lee, Matthew Vyas, Apoorv Tjandra, Andros Luthra, Mahi Guo, Baishan Wang, Huiyu Afouras, Triantafyllos Kant, David Hsu, Wei-Ning |
| author_facet | Prajwal, K R Shi, Bowen Lee, Matthew Vyas, Apoorv Tjandra, Andros Luthra, Mahi Guo, Baishan Wang, Huiyu Afouras, Triantafyllos Kant, David Hsu, Wei-Ning |
| contents | We introduce MusicFlow, a cascaded text-to-music generation model based on flow matching. Based on self-supervised representations to bridge between text descriptions and music audios, we construct two flow matching networks to model the conditional distribution of semantic and acoustic features. Additionally, we leverage masked prediction as the training objective, enabling the model to generalize to other tasks such as music infilling and continuation in a zero-shot manner. Experiments on MusicCaps reveal that the music generated by MusicFlow exhibits superior quality and text coherence despite being over $2\sim5$ times smaller and requiring $5$ times fewer iterative steps. Simultaneously, the model can perform other music generation tasks and achieves competitive performance in music infilling and continuation. Our code and model will be publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_20478 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MusicFlow: Cascaded Flow Matching for Text Guided Music Generation Prajwal, K R Shi, Bowen Lee, Matthew Vyas, Apoorv Tjandra, Andros Luthra, Mahi Guo, Baishan Wang, Huiyu Afouras, Triantafyllos Kant, David Hsu, Wei-Ning Sound Artificial Intelligence Audio and Speech Processing We introduce MusicFlow, a cascaded text-to-music generation model based on flow matching. Based on self-supervised representations to bridge between text descriptions and music audios, we construct two flow matching networks to model the conditional distribution of semantic and acoustic features. Additionally, we leverage masked prediction as the training objective, enabling the model to generalize to other tasks such as music infilling and continuation in a zero-shot manner. Experiments on MusicCaps reveal that the music generated by MusicFlow exhibits superior quality and text coherence despite being over $2\sim5$ times smaller and requiring $5$ times fewer iterative steps. Simultaneously, the model can perform other music generation tasks and achieves competitive performance in music infilling and continuation. Our code and model will be publicly available. |
| title | MusicFlow: Cascaded Flow Matching for Text Guided Music Generation |
| topic | Sound Artificial Intelligence Audio and Speech Processing |
| url | https://arxiv.org/abs/2410.20478 |