Echoes Over Time: Unlocking Length Generalization in Video-to-Audio Generation Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913031818051584 |
|---|---|
| author | Simon, Christian Ishii, Masato Wang, Wei-Yao Saito, Koichi Hayakawa, Akio Shim, Dongseok Zhong, Zhi Cui, Shuyang Takahashi, Shusuke Shibuya, Takashi Mitsufuji, Yuki |
| author_facet | Simon, Christian Ishii, Masato Wang, Wei-Yao Saito, Koichi Hayakawa, Akio Shim, Dongseok Zhong, Zhi Cui, Shuyang Takahashi, Shusuke Shibuya, Takashi Mitsufuji, Yuki |
| contents | Scaling multimodal alignment between video and audio is challenging, particularly due to limited data and the mismatch between text descriptions and frame-level video information. In this work, we tackle the scaling challenge in multimodal-to-audio generation, examining whether models trained on short instances can generalize to longer ones during testing. To tackle this challenge, we present multimodal hierarchical networks so-called MMHNet, an enhanced extension of state-of-the-art video-to-audio models. Our approach integrates a hierarchical method and non-causal Mamba to support long-form audio generation. Our proposed method significantly improves long audio generation up to more than 5 minutes. We also prove that training short and testing long is possible in the video-to-audio generation tasks without training on the longer durations. We show in our experiments that our proposed method could achieve remarkable results on long-video to audio benchmarks, beating prior works in video-to-audio tasks. Moreover, we showcase our model capability in generating more than 5 minutes, while prior video-to-audio methods fall short in generating with long durations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_20981 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Echoes Over Time: Unlocking Length Generalization in Video-to-Audio Generation Models Simon, Christian Ishii, Masato Wang, Wei-Yao Saito, Koichi Hayakawa, Akio Shim, Dongseok Zhong, Zhi Cui, Shuyang Takahashi, Shusuke Shibuya, Takashi Mitsufuji, Yuki Computer Vision and Pattern Recognition Artificial Intelligence Scaling multimodal alignment between video and audio is challenging, particularly due to limited data and the mismatch between text descriptions and frame-level video information. In this work, we tackle the scaling challenge in multimodal-to-audio generation, examining whether models trained on short instances can generalize to longer ones during testing. To tackle this challenge, we present multimodal hierarchical networks so-called MMHNet, an enhanced extension of state-of-the-art video-to-audio models. Our approach integrates a hierarchical method and non-causal Mamba to support long-form audio generation. Our proposed method significantly improves long audio generation up to more than 5 minutes. We also prove that training short and testing long is possible in the video-to-audio generation tasks without training on the longer durations. We show in our experiments that our proposed method could achieve remarkable results on long-video to audio benchmarks, beating prior works in video-to-audio tasks. Moreover, we showcase our model capability in generating more than 5 minutes, while prior video-to-audio methods fall short in generating with long durations. |
| title | Echoes Over Time: Unlocking Length Generalization in Video-to-Audio Generation Models |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2602.20981 |