VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912364057591808 |
|---|---|
| author | Tian, Zeyue Liu, Zhaoyang Yuan, Ruibin Pan, Jiahao Liu, Qifeng Tan, Xu Chen, Qifeng Xue, Wei Guo, Yike |
| author_facet | Tian, Zeyue Liu, Zhaoyang Yuan, Ruibin Pan, Jiahao Liu, Qifeng Tan, Xu Chen, Qifeng Xue, Wei Guo, Yike |
| contents | In this work, we systematically study music generation conditioned solely on the video. First, we present a large-scale dataset comprising 360K video-music pairs, including various genres such as movie trailers, advertisements, and documentaries. Furthermore, we propose VidMuse, a simple framework for generating music aligned with video inputs. VidMuse stands out by producing high-fidelity music that is both acoustically and semantically aligned with the video. By incorporating local and global visual cues, VidMuse enables the creation of musically coherent audio tracks that consistently match the video content through Long-Short-Term modeling. Through extensive experiments, VidMuse outperforms existing models in terms of audio quality, diversity, and audio-visual alignment. The code and datasets are available at https://vidmuse.github.io/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_04321 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling Tian, Zeyue Liu, Zhaoyang Yuan, Ruibin Pan, Jiahao Liu, Qifeng Tan, Xu Chen, Qifeng Xue, Wei Guo, Yike Computer Vision and Pattern Recognition Machine Learning Multimedia Sound In this work, we systematically study music generation conditioned solely on the video. First, we present a large-scale dataset comprising 360K video-music pairs, including various genres such as movie trailers, advertisements, and documentaries. Furthermore, we propose VidMuse, a simple framework for generating music aligned with video inputs. VidMuse stands out by producing high-fidelity music that is both acoustically and semantically aligned with the video. By incorporating local and global visual cues, VidMuse enables the creation of musically coherent audio tracks that consistently match the video content through Long-Short-Term modeling. Through extensive experiments, VidMuse outperforms existing models in terms of audio quality, diversity, and audio-visual alignment. The code and datasets are available at https://vidmuse.github.io/. |
| title | VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling |
| topic | Computer Vision and Pattern Recognition Machine Learning Multimedia Sound |
| url | https://arxiv.org/abs/2406.04321 |