DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image Restoration Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908738783281152 |
|---|---|
| author | Yeh, Chang-Han Shiu, Hau-Shiang Lin, Chin-Yang Wang, Zhixiang Hsiao, Chi-Wei Chen, Ting-Hsuan Liu, Yu-Lun |
| author_facet | Yeh, Chang-Han Shiu, Hau-Shiang Lin, Chin-Yang Wang, Zhixiang Hsiao, Chi-Wei Chen, Ting-Hsuan Liu, Yu-Lun |
| contents | We present DiffIR2VR-Zero, a zero-shot framework that enables any pre-trained image restoration diffusion model to perform high-quality video restoration without additional training. While image diffusion models have shown remarkable restoration capabilities, their direct application to video leads to temporal inconsistencies, and existing video restoration methods require extensive retraining for different degradation types. Our approach addresses these challenges through two key innovations: a hierarchical latent warping strategy that maintains consistency across both keyframes and local frames, and a hybrid token merging mechanism that adaptively combines optical flow and feature matching. Through extensive experiments, we demonstrate that our method not only maintains the high-quality restoration of base diffusion models but also achieves superior temporal consistency across diverse datasets and degradation conditions, including challenging scenarios like 8$\times$ super-resolution and severe noise. Importantly, our framework works with any image restoration diffusion model, providing a versatile solution for video enhancement without task-specific training or modifications. Project page: https://jimmycv07.github.io/DiffIR2VR_web/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_01519 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image Restoration Models Yeh, Chang-Han Shiu, Hau-Shiang Lin, Chin-Yang Wang, Zhixiang Hsiao, Chi-Wei Chen, Ting-Hsuan Liu, Yu-Lun Computer Vision and Pattern Recognition We present DiffIR2VR-Zero, a zero-shot framework that enables any pre-trained image restoration diffusion model to perform high-quality video restoration without additional training. While image diffusion models have shown remarkable restoration capabilities, their direct application to video leads to temporal inconsistencies, and existing video restoration methods require extensive retraining for different degradation types. Our approach addresses these challenges through two key innovations: a hierarchical latent warping strategy that maintains consistency across both keyframes and local frames, and a hybrid token merging mechanism that adaptively combines optical flow and feature matching. Through extensive experiments, we demonstrate that our method not only maintains the high-quality restoration of base diffusion models but also achieves superior temporal consistency across diverse datasets and degradation conditions, including challenging scenarios like 8$\times$ super-resolution and severe noise. Importantly, our framework works with any image restoration diffusion model, providing a versatile solution for video enhancement without task-specific training or modifications. Project page: https://jimmycv07.github.io/DiffIR2VR_web/ |
| title | DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image Restoration Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.01519 |