Mind the Time: Temporally-Controlled Multi-Event Video Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929748262780928 |
|---|---|
| author | Wu, Ziyi Siarohin, Aliaksandr Menapace, Willi Skorokhodov, Ivan Fang, Yuwei Chordia, Varnith Gilitschenski, Igor Tulyakov, Sergey |
| author_facet | Wu, Ziyi Siarohin, Aliaksandr Menapace, Willi Skorokhodov, Ivan Fang, Yuwei Chordia, Varnith Gilitschenski, Igor Tulyakov, Sergey |
| contents | Real-world videos consist of sequences of events. Generating such sequences with precise temporal control is infeasible with existing video generators that rely on a single paragraph of text as input. When tasked with generating multiple events described using a single prompt, such methods often ignore some of the events or fail to arrange them in the correct order. To address this limitation, we present MinT, a multi-event video generator with temporal control. Our key insight is to bind each event to a specific period in the generated video, which allows the model to focus on one event at a time. To enable time-aware interactions between event captions and video tokens, we design a time-based positional encoding method, dubbed ReRoPE. This encoding helps to guide the cross-attention operation. By fine-tuning a pre-trained video diffusion transformer on temporally grounded data, our approach produces coherent videos with smoothly connected events. For the first time in the literature, our model offers control over the timing of events in generated videos. Extensive experiments demonstrate that MinT outperforms existing commercial and open-source models by a large margin. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_05263 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Mind the Time: Temporally-Controlled Multi-Event Video Generation Wu, Ziyi Siarohin, Aliaksandr Menapace, Willi Skorokhodov, Ivan Fang, Yuwei Chordia, Varnith Gilitschenski, Igor Tulyakov, Sergey Computer Vision and Pattern Recognition Real-world videos consist of sequences of events. Generating such sequences with precise temporal control is infeasible with existing video generators that rely on a single paragraph of text as input. When tasked with generating multiple events described using a single prompt, such methods often ignore some of the events or fail to arrange them in the correct order. To address this limitation, we present MinT, a multi-event video generator with temporal control. Our key insight is to bind each event to a specific period in the generated video, which allows the model to focus on one event at a time. To enable time-aware interactions between event captions and video tokens, we design a time-based positional encoding method, dubbed ReRoPE. This encoding helps to guide the cross-attention operation. By fine-tuning a pre-trained video diffusion transformer on temporally grounded data, our approach produces coherent videos with smoothly connected events. For the first time in the literature, our model offers control over the timing of events in generated videos. Extensive experiments demonstrate that MinT outperforms existing commercial and open-source models by a large margin. |
| title | Mind the Time: Temporally-Controlled Multi-Event Video Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.05263 |