GVMGen: A General Video-to-Music Generation Model with Hierarchical Attentions
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916695363289088 |
|---|---|
| author | Zuo, Heda You, Weitao Wu, Junxian Ren, Shihong Chen, Pei Zhou, Mingxu Lu, Yujia Sun, Lingyun |
| author_facet | Zuo, Heda You, Weitao Wu, Junxian Ren, Shihong Chen, Pei Zhou, Mingxu Lu, Yujia Sun, Lingyun |
| contents | Composing music for video is essential yet challenging, leading to a growing interest in automating music generation for video applications. Existing approaches often struggle to achieve robust music-video correspondence and generative diversity, primarily due to inadequate feature alignment methods and insufficient datasets. In this study, we present General Video-to-Music Generation model (GVMGen), designed for generating high-related music to the video input. Our model employs hierarchical attentions to extract and align video features with music in both spatial and temporal dimensions, ensuring the preservation of pertinent features while minimizing redundancy. Remarkably, our method is versatile, capable of generating multi-style music from different video inputs, even in zero-shot scenarios. We also propose an evaluation model along with two novel objective metrics for assessing video-music alignment. Additionally, we have compiled a large-scale dataset comprising diverse types of video-music pairs. Experimental results demonstrate that GVMGen surpasses previous models in terms of music-video correspondence, generative diversity, and application universality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_09972 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | GVMGen: A General Video-to-Music Generation Model with Hierarchical Attentions Zuo, Heda You, Weitao Wu, Junxian Ren, Shihong Chen, Pei Zhou, Mingxu Lu, Yujia Sun, Lingyun Sound Artificial Intelligence Multimedia Audio and Speech Processing Composing music for video is essential yet challenging, leading to a growing interest in automating music generation for video applications. Existing approaches often struggle to achieve robust music-video correspondence and generative diversity, primarily due to inadequate feature alignment methods and insufficient datasets. In this study, we present General Video-to-Music Generation model (GVMGen), designed for generating high-related music to the video input. Our model employs hierarchical attentions to extract and align video features with music in both spatial and temporal dimensions, ensuring the preservation of pertinent features while minimizing redundancy. Remarkably, our method is versatile, capable of generating multi-style music from different video inputs, even in zero-shot scenarios. We also propose an evaluation model along with two novel objective metrics for assessing video-music alignment. Additionally, we have compiled a large-scale dataset comprising diverse types of video-music pairs. Experimental results demonstrate that GVMGen surpasses previous models in terms of music-video correspondence, generative diversity, and application universality. |
| title | GVMGen: A General Video-to-Music Generation Model with Hierarchical Attentions |
| topic | Sound Artificial Intelligence Multimedia Audio and Speech Processing |
| url | https://arxiv.org/abs/2501.09972 |