Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2407.03648 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913548596150272 |
|---|---|
| author | Lan, Gael Le Shi, Bowen Ni, Zhaoheng Srinivasan, Sidd Kumar, Anurag Ellis, Brian Kant, David Nagaraja, Varun Chang, Ernie Hsu, Wei-Ning Shi, Yangyang Chandra, Vikas |
| author_facet | Lan, Gael Le Shi, Bowen Ni, Zhaoheng Srinivasan, Sidd Kumar, Anurag Ellis, Brian Kant, David Nagaraja, Varun Chang, Ernie Hsu, Wei-Ning Shi, Yangyang Chandra, Vikas |
| contents | We introduce MelodyFlow, an efficient text-controllable high-fidelity music generation and editing model. It operates on continuous latent representations from a low frame rate 48 kHz stereo variational auto encoder codec. Based on a diffusion transformer architecture trained on a flow-matching objective the model can edit diverse high quality stereo samples of variable duration, with simple text descriptions. We adapt the ReNoise latent inversion method to flow matching and compare it with the original implementation and naive denoising diffusion implicit model (DDIM) inversion on a variety of music editing prompts. Our results indicate that our latent inversion outperforms both ReNoise and DDIM for zero-shot test-time text-guided editing on several objective metrics. Subjective evaluations exhibit a substantial improvement over previous state of the art for music editing. Code and model weights will be publicly made available. Samples are available at https://melodyflow.github.io. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_03648 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | High Fidelity Text-Guided Music Editing via Single-Stage Flow Matching Lan, Gael Le Shi, Bowen Ni, Zhaoheng Srinivasan, Sidd Kumar, Anurag Ellis, Brian Kant, David Nagaraja, Varun Chang, Ernie Hsu, Wei-Ning Shi, Yangyang Chandra, Vikas Audio and Speech Processing Sound We introduce MelodyFlow, an efficient text-controllable high-fidelity music generation and editing model. It operates on continuous latent representations from a low frame rate 48 kHz stereo variational auto encoder codec. Based on a diffusion transformer architecture trained on a flow-matching objective the model can edit diverse high quality stereo samples of variable duration, with simple text descriptions. We adapt the ReNoise latent inversion method to flow matching and compare it with the original implementation and naive denoising diffusion implicit model (DDIM) inversion on a variety of music editing prompts. Our results indicate that our latent inversion outperforms both ReNoise and DDIM for zero-shot test-time text-guided editing on several objective metrics. Subjective evaluations exhibit a substantial improvement over previous state of the art for music editing. Code and model weights will be publicly made available. Samples are available at https://melodyflow.github.io. |
| title | High Fidelity Text-Guided Music Editing via Single-Stage Flow Matching |
| topic | Audio and Speech Processing Sound |
| url | https://arxiv.org/abs/2407.03648 |