Semantically consistent Video-to-Audio Generation using Multimodal Language Large Model
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917651062718464 |
|---|---|
| author | Chen, Gehui Wang, Guan'an Huang, Xiaowen Sang, Jitao |
| author_facet | Chen, Gehui Wang, Guan'an Huang, Xiaowen Sang, Jitao |
| contents | Existing works have made strides in video generation, but the lack of sound effects (SFX) and background music (BGM) hinders a complete and immersive viewer experience. We introduce a novel semantically consistent v ideo-to-audio generation framework, namely SVA, which automatically generates audio semantically consistent with the given video content. The framework harnesses the power of multimodal large language model (MLLM) to understand video semantics from a key frame and generate creative audio schemes, which are then utilized as prompts for text-to-audio models, resulting in video-to-audio generation with natural language as an interface. We show the satisfactory performance of SVA through case study and discuss the limitations along with the future research direction. The project page is available at https://huiz-a.github.io/audio4video.github.io/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_16305 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Semantically consistent Video-to-Audio Generation using Multimodal Language Large Model Chen, Gehui Wang, Guan'an Huang, Xiaowen Sang, Jitao Multimedia Sound Audio and Speech Processing Existing works have made strides in video generation, but the lack of sound effects (SFX) and background music (BGM) hinders a complete and immersive viewer experience. We introduce a novel semantically consistent v ideo-to-audio generation framework, namely SVA, which automatically generates audio semantically consistent with the given video content. The framework harnesses the power of multimodal large language model (MLLM) to understand video semantics from a key frame and generate creative audio schemes, which are then utilized as prompts for text-to-audio models, resulting in video-to-audio generation with natural language as an interface. We show the satisfactory performance of SVA through case study and discuss the limitations along with the future research direction. The project page is available at https://huiz-a.github.io/audio4video.github.io/. |
| title | Semantically consistent Video-to-Audio Generation using Multimodal Language Large Model |
| topic | Multimedia Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2404.16305 |