VIA: Unified Spatiotemporal Video Adaptation Framework for Global and Local Video Editing
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913761488535552 |
|---|---|
| author | Gu, Jing Fang, Yuwei Skorokhodov, Ivan Wonka, Peter Du, Xinya Tulyakov, Sergey Wang, Xin Eric |
| author_facet | Gu, Jing Fang, Yuwei Skorokhodov, Ivan Wonka, Peter Du, Xinya Tulyakov, Sergey Wang, Xin Eric |
| contents | Video editing serves as a fundamental pillar of digital media, spanning applications in entertainment, education, and professional communication. However, previous methods often overlook the necessity of comprehensively understanding both global and local contexts, leading to inaccurate and inconsistent edits in the spatiotemporal dimension, especially for long videos. In this paper, we introduce VIA, a unified spatiotemporal Video Adaptation framework for global and local video editing, pushing the limits of consistently editing minute-long videos. First, to ensure local consistency within individual frames, we designed test-time editing adaptation to adapt a pre-trained image editing model for improving consistency between potential editing directions and the text instruction, and adapts masked latent variables for precise local control. Furthermore, to maintain global consistency over the video sequence, we introduce spatiotemporal adaptation that recursively gather consistent attention variables in key frames and strategically applies them across the whole sequence to realize the editing effects. Extensive experiments demonstrate that, compared to baseline methods, our VIA approach produces edits that are more faithful to the source videos, more coherent in the spatiotemporal context, and more precise in local control. More importantly, we show that VIA can achieve consistent long video editing in minutes, unlocking the potential for advanced video editing tasks over long video sequences. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_12831 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | VIA: Unified Spatiotemporal Video Adaptation Framework for Global and Local Video Editing Gu, Jing Fang, Yuwei Skorokhodov, Ivan Wonka, Peter Du, Xinya Tulyakov, Sergey Wang, Xin Eric Computer Vision and Pattern Recognition Artificial Intelligence Multimedia Video editing serves as a fundamental pillar of digital media, spanning applications in entertainment, education, and professional communication. However, previous methods often overlook the necessity of comprehensively understanding both global and local contexts, leading to inaccurate and inconsistent edits in the spatiotemporal dimension, especially for long videos. In this paper, we introduce VIA, a unified spatiotemporal Video Adaptation framework for global and local video editing, pushing the limits of consistently editing minute-long videos. First, to ensure local consistency within individual frames, we designed test-time editing adaptation to adapt a pre-trained image editing model for improving consistency between potential editing directions and the text instruction, and adapts masked latent variables for precise local control. Furthermore, to maintain global consistency over the video sequence, we introduce spatiotemporal adaptation that recursively gather consistent attention variables in key frames and strategically applies them across the whole sequence to realize the editing effects. Extensive experiments demonstrate that, compared to baseline methods, our VIA approach produces edits that are more faithful to the source videos, more coherent in the spatiotemporal context, and more precise in local control. More importantly, we show that VIA can achieve consistent long video editing in minutes, unlocking the potential for advanced video editing tasks over long video sequences. |
| title | VIA: Unified Spatiotemporal Video Adaptation Framework for Global and Local Video Editing |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Multimedia |
| url | https://arxiv.org/abs/2406.12831 |