Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2312.12635 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916591110717440 |
|---|---|
| author | Jin, Shutong Wang, Ruiyu Pokorny, Florian T. |
| author_facet | Jin, Shutong Wang, Ruiyu Pokorny, Florian T. |
| contents | Even though large-scale text-to-image generative models show promising performance in synthesizing high-quality images, applying these models directly to image editing remains a significant challenge. This challenge is further amplified in video editing due to the additional dimension of time. This is especially the case for editing real-world videos as it necessitates maintaining a stable structural layout across frames while executing localized edits without disrupting the existing content. In this paper, we propose RealCraft, an attention-control-based method for zero-shot real-world video editing. By swapping cross-attention for new feature injection and relaxing spatial-temporal attention of the editing object, we achieve localized shape-wise edit along with enhanced temporal consistency. Our model directly uses Stable Diffusion and operates without the need for additional information. We showcase the proposed zero-shot attention-control-based method across a range of videos, demonstrating shape-wise, time-consistent and parameter-free editing in videos of up to 64 frames. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_12635 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | RealCraft: Attention Control as A Tool for Zero-Shot Consistent Video Editing Jin, Shutong Wang, Ruiyu Pokorny, Florian T. Computer Vision and Pattern Recognition Even though large-scale text-to-image generative models show promising performance in synthesizing high-quality images, applying these models directly to image editing remains a significant challenge. This challenge is further amplified in video editing due to the additional dimension of time. This is especially the case for editing real-world videos as it necessitates maintaining a stable structural layout across frames while executing localized edits without disrupting the existing content. In this paper, we propose RealCraft, an attention-control-based method for zero-shot real-world video editing. By swapping cross-attention for new feature injection and relaxing spatial-temporal attention of the editing object, we achieve localized shape-wise edit along with enhanced temporal consistency. Our model directly uses Stable Diffusion and operates without the need for additional information. We showcase the proposed zero-shot attention-control-based method across a range of videos, demonstrating shape-wise, time-consistent and parameter-free editing in videos of up to 64 frames. |
| title | RealCraft: Attention Control as A Tool for Zero-Shot Consistent Video Editing |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2312.12635 |