MuMu-LLaMA: Multi-modal Music Understanding and Generation via Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913603193405440 |
|---|---|
| author | Liu, Shansong Hussain, Atin Sakkeer Wu, Qilong Sun, Chenshuo Shan, Ying |
| author_facet | Liu, Shansong Hussain, Atin Sakkeer Wu, Qilong Sun, Chenshuo Shan, Ying |
| contents | Research on large language models has advanced significantly across text, speech, images, and videos. However, multi-modal music understanding and generation remain underexplored due to the lack of well-annotated datasets. To address this, we introduce a dataset with 167.69 hours of multi-modal data, including text, images, videos, and music annotations. Based on this dataset, we propose MuMu-LLaMA, a model that leverages pre-trained encoders for music, images, and videos. For music generation, we integrate AudioLDM 2 and MusicGen. Our evaluation across four tasks--music understanding, text-to-music generation, prompt-based music editing, and multi-modal music generation--demonstrates that MuMu-LLaMA outperforms state-of-the-art models, showing its potential for multi-modal music applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_06660 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MuMu-LLaMA: Multi-modal Music Understanding and Generation via Large Language Models Liu, Shansong Hussain, Atin Sakkeer Wu, Qilong Sun, Chenshuo Shan, Ying Sound Multimedia Audio and Speech Processing Research on large language models has advanced significantly across text, speech, images, and videos. However, multi-modal music understanding and generation remain underexplored due to the lack of well-annotated datasets. To address this, we introduce a dataset with 167.69 hours of multi-modal data, including text, images, videos, and music annotations. Based on this dataset, we propose MuMu-LLaMA, a model that leverages pre-trained encoders for music, images, and videos. For music generation, we integrate AudioLDM 2 and MusicGen. Our evaluation across four tasks--music understanding, text-to-music generation, prompt-based music editing, and multi-modal music generation--demonstrates that MuMu-LLaMA outperforms state-of-the-art models, showing its potential for multi-modal music applications. |
| title | MuMu-LLaMA: Multi-modal Music Understanding and Generation via Large Language Models |
| topic | Sound Multimedia Audio and Speech Processing |
| url | https://arxiv.org/abs/2412.06660 |