DeepResonance: Enhancing Multimodal Music Understanding via Music-centric Multi-way Instruction Tuning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911170251718656 |
|---|---|
| author | Mao, Zhuoyuan Zhao, Mengjie Wu, Qiyu Wakaki, Hiromi Mitsufuji, Yuki |
| author_facet | Mao, Zhuoyuan Zhao, Mengjie Wu, Qiyu Wakaki, Hiromi Mitsufuji, Yuki |
| contents | Recent advancements in music large language models (LLMs) have significantly improved music understanding tasks, which involve the model's ability to analyze and interpret various musical elements. These improvements primarily focused on integrating both music and text inputs. However, the potential of incorporating additional modalities such as images, videos and textual music features to enhance music understanding remains unexplored. To bridge this gap, we propose DeepResonance, a multimodal music understanding LLM fine-tuned via multi-way instruction tuning with multi-way aligned music, text, image, and video data. To this end, we construct Music4way-MI2T, Music4way-MV2T, and Music4way-Any2T, three 4-way training and evaluation datasets designed to enable DeepResonance to integrate both visual and textual music feature content. We also introduce multi-sampled ImageBind embeddings and a pre-LLM fusion Transformer to enhance modality fusion prior to input into text LLMs, tailoring for multi-way instruction tuning. Our model achieves state-of-the-art performances across six music understanding tasks, highlighting the benefits of the auxiliary modalities and the structural superiority of DeepResonance. We open-source the codes, models and datasets we constructed: github.com/sony/DeepResonance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_12623 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DeepResonance: Enhancing Multimodal Music Understanding via Music-centric Multi-way Instruction Tuning Mao, Zhuoyuan Zhao, Mengjie Wu, Qiyu Wakaki, Hiromi Mitsufuji, Yuki Sound Artificial Intelligence Computation and Language Multimedia Audio and Speech Processing Recent advancements in music large language models (LLMs) have significantly improved music understanding tasks, which involve the model's ability to analyze and interpret various musical elements. These improvements primarily focused on integrating both music and text inputs. However, the potential of incorporating additional modalities such as images, videos and textual music features to enhance music understanding remains unexplored. To bridge this gap, we propose DeepResonance, a multimodal music understanding LLM fine-tuned via multi-way instruction tuning with multi-way aligned music, text, image, and video data. To this end, we construct Music4way-MI2T, Music4way-MV2T, and Music4way-Any2T, three 4-way training and evaluation datasets designed to enable DeepResonance to integrate both visual and textual music feature content. We also introduce multi-sampled ImageBind embeddings and a pre-LLM fusion Transformer to enhance modality fusion prior to input into text LLMs, tailoring for multi-way instruction tuning. Our model achieves state-of-the-art performances across six music understanding tasks, highlighting the benefits of the auxiliary modalities and the structural superiority of DeepResonance. We open-source the codes, models and datasets we constructed: github.com/sony/DeepResonance. |
| title | DeepResonance: Enhancing Multimodal Music Understanding via Music-centric Multi-way Instruction Tuning |
| topic | Sound Artificial Intelligence Computation and Language Multimedia Audio and Speech Processing |
| url | https://arxiv.org/abs/2502.12623 |