MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916188305489920 |
|---|---|
| author | Ma, Haoyu Mahdizadehaghdam, Shahin Wu, Bichen Fan, Zhipeng Gu, Yuchao Zhao, Wenliang Shapira, Lior Xie, Xiaohui |
| author_facet | Ma, Haoyu Mahdizadehaghdam, Shahin Wu, Bichen Fan, Zhipeng Gu, Yuchao Zhao, Wenliang Shapira, Lior Xie, Xiaohui |
| contents | Recent advances in generative AI have significantly enhanced image and video editing, particularly in the context of text prompt control. State-of-the-art approaches predominantly rely on diffusion models to accomplish these tasks. However, the computational demands of diffusion-based methods are substantial, often necessitating large-scale paired datasets for training, and therefore challenging the deployment in real applications. To address these issues, this paper breaks down the text-based video editing task into two stages. First, we leverage an pre-trained text-to-image diffusion model to simultaneously edit few keyframes in an zero-shot way. Second, we introduce an efficient model called MaskINT, which is built on non-autoregressive masked generative transformers and specializes in frame interpolation between the edited keyframes, using the structural guidance from intermediate frames. Experimental results suggest that our MaskINT achieves comparable performance with diffusion-based methodologies, while significantly improve the inference time. This research offers a practical solution for text-based video editing and showcases the potential of non-autoregressive masked generative transformers in this domain. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_12468 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers Ma, Haoyu Mahdizadehaghdam, Shahin Wu, Bichen Fan, Zhipeng Gu, Yuchao Zhao, Wenliang Shapira, Lior Xie, Xiaohui Computer Vision and Pattern Recognition Recent advances in generative AI have significantly enhanced image and video editing, particularly in the context of text prompt control. State-of-the-art approaches predominantly rely on diffusion models to accomplish these tasks. However, the computational demands of diffusion-based methods are substantial, often necessitating large-scale paired datasets for training, and therefore challenging the deployment in real applications. To address these issues, this paper breaks down the text-based video editing task into two stages. First, we leverage an pre-trained text-to-image diffusion model to simultaneously edit few keyframes in an zero-shot way. Second, we introduce an efficient model called MaskINT, which is built on non-autoregressive masked generative transformers and specializes in frame interpolation between the edited keyframes, using the structural guidance from intermediate frames. Experimental results suggest that our MaskINT achieves comparable performance with diffusion-based methodologies, while significantly improve the inference time. This research offers a practical solution for text-based video editing and showcases the potential of non-autoregressive masked generative transformers in this domain. |
| title | MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2312.12468 |