Cross-Modal Learning for Music-to-Music-Video Description Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913736043790336 |
|---|---|
| author | Mao, Zhuoyuan Zhao, Mengjie Wu, Qiyu Zhong, Zhi Liao, Wei-Hsiang Wakaki, Hiromi Mitsufuji, Yuki |
| author_facet | Mao, Zhuoyuan Zhao, Mengjie Wu, Qiyu Zhong, Zhi Liao, Wei-Hsiang Wakaki, Hiromi Mitsufuji, Yuki |
| contents | Music-to-music-video generation is a challenging task due to the intrinsic differences between the music and video modalities. The advent of powerful text-to-video diffusion models has opened a promising pathway for music-video (MV) generation by first addressing the music-to-MV description task and subsequently leveraging these models for video generation. In this study, we focus on the MV description generation task and propose a comprehensive pipeline encompassing training data construction and multimodal model fine-tuning. We fine-tune existing pre-trained multimodal models on our newly constructed music-to-MV description dataset based on the Music4All dataset, which integrates both musical and visual information. Our experimental results demonstrate that music representations can be effectively mapped to textual domains, enabling the generation of meaningful MV description directly from music inputs. We also identify key components in the dataset construction pipeline that critically impact the quality of MV description and highlight specific musical attributes that warrant greater focus for improved MV description generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_11190 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Cross-Modal Learning for Music-to-Music-Video Description Generation Mao, Zhuoyuan Zhao, Mengjie Wu, Qiyu Zhong, Zhi Liao, Wei-Hsiang Wakaki, Hiromi Mitsufuji, Yuki Sound Artificial Intelligence Computation and Language Multimedia Audio and Speech Processing Music-to-music-video generation is a challenging task due to the intrinsic differences between the music and video modalities. The advent of powerful text-to-video diffusion models has opened a promising pathway for music-video (MV) generation by first addressing the music-to-MV description task and subsequently leveraging these models for video generation. In this study, we focus on the MV description generation task and propose a comprehensive pipeline encompassing training data construction and multimodal model fine-tuning. We fine-tune existing pre-trained multimodal models on our newly constructed music-to-MV description dataset based on the Music4All dataset, which integrates both musical and visual information. Our experimental results demonstrate that music representations can be effectively mapped to textual domains, enabling the generation of meaningful MV description directly from music inputs. We also identify key components in the dataset construction pipeline that critically impact the quality of MV description and highlight specific musical attributes that warrant greater focus for improved MV description generation. |
| title | Cross-Modal Learning for Music-to-Music-Video Description Generation |
| topic | Sound Artificial Intelligence Computation and Language Multimedia Audio and Speech Processing |
| url | https://arxiv.org/abs/2503.11190 |