Enhancing Low-Cost Video Editing with Lightweight Adaptors and Temporal-Aware Inversion
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908402914951168 |
|---|---|
| author | He, Yangfan Li, Sida Wang, Jianhui Li, Kun Song, Xinyuan Yuan, Xinhang Li, Keqin Lu, Kuan Huo, Menghao Tang, Jingqun Xin, Yi Chen, Jiaqi Zhang, Miao Wang, Xueqian |
| author_facet | He, Yangfan Li, Sida Wang, Jianhui Li, Kun Song, Xinyuan Yuan, Xinhang Li, Keqin Lu, Kuan Huo, Menghao Tang, Jingqun Xin, Yi Chen, Jiaqi Zhang, Miao Wang, Xueqian |
| contents | Recent advancements in text-to-image (T2I) generation using diffusion models have enabled cost-effective video-editing applications by leveraging pre-trained models, eliminating the need for resource-intensive training. However, the frame-independence of T2I generation often results in poor temporal consistency. Existing methods address this issue through temporal layer fine-tuning or inference-based temporal propagation, but these approaches suffer from high training costs or limited temporal coherence. To address these challenges, we propose a General and Efficient Adapter (GE-Adapter) that integrates temporal-spatial and semantic consistency with Baliteral DDIM inversion. This framework introduces three key components: (1) Frame-based Temporal Consistency Blocks (FTC Blocks) to capture frame-specific features and enforce smooth inter-frame transitions via temporally-aware loss functions; (2) Channel-dependent Spatial Consistency Blocks (SCD Blocks) employing bilateral filters to enhance spatial coherence by reducing noise and artifacts; and (3) Token-based Semantic Consistency Module (TSC Module) to maintain semantic alignment using shared prompt tokens and frame-specific tokens. Our method significantly improves perceptual quality, text-image alignment, and temporal coherence, as demonstrated on the MSR-VTT dataset. Additionally, it achieves enhanced fidelity and frame-to-frame coherence, offering a practical solution for T2V editing. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_04606 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Enhancing Low-Cost Video Editing with Lightweight Adaptors and Temporal-Aware Inversion He, Yangfan Li, Sida Wang, Jianhui Li, Kun Song, Xinyuan Yuan, Xinhang Li, Keqin Lu, Kuan Huo, Menghao Tang, Jingqun Xin, Yi Chen, Jiaqi Zhang, Miao Wang, Xueqian Computer Vision and Pattern Recognition Recent advancements in text-to-image (T2I) generation using diffusion models have enabled cost-effective video-editing applications by leveraging pre-trained models, eliminating the need for resource-intensive training. However, the frame-independence of T2I generation often results in poor temporal consistency. Existing methods address this issue through temporal layer fine-tuning or inference-based temporal propagation, but these approaches suffer from high training costs or limited temporal coherence. To address these challenges, we propose a General and Efficient Adapter (GE-Adapter) that integrates temporal-spatial and semantic consistency with Baliteral DDIM inversion. This framework introduces three key components: (1) Frame-based Temporal Consistency Blocks (FTC Blocks) to capture frame-specific features and enforce smooth inter-frame transitions via temporally-aware loss functions; (2) Channel-dependent Spatial Consistency Blocks (SCD Blocks) employing bilateral filters to enhance spatial coherence by reducing noise and artifacts; and (3) Token-based Semantic Consistency Module (TSC Module) to maintain semantic alignment using shared prompt tokens and frame-specific tokens. Our method significantly improves perceptual quality, text-image alignment, and temporal coherence, as demonstrated on the MSR-VTT dataset. Additionally, it achieves enhanced fidelity and frame-to-frame coherence, offering a practical solution for T2V editing. |
| title | Enhancing Low-Cost Video Editing with Lightweight Adaptors and Temporal-Aware Inversion |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2501.04606 |