Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.08307 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915193958694912 |
|---|---|
| author | Ergasti, Alex Tarollo, Giuseppe Gabriele Botti, Filippo Fontanini, Tomaso Ferrari, Claudio Bertozzi, Massimo Prati, Andrea |
| author_facet | Ergasti, Alex Tarollo, Giuseppe Gabriele Botti, Filippo Fontanini, Tomaso Ferrari, Claudio Bertozzi, Massimo Prati, Andrea |
| contents | Joint audio-video (AV) generation is still a significant challenge in generative AI, primarily due to three critical requirements: quality of the generated samples, seamless multimodal synchronization and temporal coherence, with audio tracks that match the visual data and vice versa, and limitless video duration. In this paper, we present $^R$-FLAV, a novel transformer-based architecture that addresses all the key challenges of AV generation. We explore three distinct cross modality interaction modules, with our lightweight temporal fusion module emerging as the most effective and computationally efficient approach for aligning audio and visual modalities. Our experimental results demonstrate that $^R$-FLAV outperforms existing state-of-the-art models in multimodal AV generation tasks. Our code and checkpoints are available at https://github.com/ErgastiAlex/R-FLAV. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_08307 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | $^R$FLAV: Rolling Flow matching for infinite Audio Video generation Ergasti, Alex Tarollo, Giuseppe Gabriele Botti, Filippo Fontanini, Tomaso Ferrari, Claudio Bertozzi, Massimo Prati, Andrea Computer Vision and Pattern Recognition Joint audio-video (AV) generation is still a significant challenge in generative AI, primarily due to three critical requirements: quality of the generated samples, seamless multimodal synchronization and temporal coherence, with audio tracks that match the visual data and vice versa, and limitless video duration. In this paper, we present $^R$-FLAV, a novel transformer-based architecture that addresses all the key challenges of AV generation. We explore three distinct cross modality interaction modules, with our lightweight temporal fusion module emerging as the most effective and computationally efficient approach for aligning audio and visual modalities. Our experimental results demonstrate that $^R$-FLAV outperforms existing state-of-the-art models in multimodal AV generation tasks. Our code and checkpoints are available at https://github.com/ErgastiAlex/R-FLAV. |
| title | $^R$FLAV: Rolling Flow matching for infinite Audio Video generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2503.08307 |