High-Resolution Frame Interpolation with Patch-based Cascaded Diffusion
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909596331802624 |
|---|---|
| author | Hur, Junhwa Herrmann, Charles Saxena, Saurabh Kontkanen, Janne Lai, Wei-Sheng Shih, Yichang Rubinstein, Michael Fleet, David J. Sun, Deqing |
| author_facet | Hur, Junhwa Herrmann, Charles Saxena, Saurabh Kontkanen, Janne Lai, Wei-Sheng Shih, Yichang Rubinstein, Michael Fleet, David J. Sun, Deqing |
| contents | Despite the recent progress, existing frame interpolation methods still struggle with processing extremely high resolution input and handling challenging cases such as repetitive textures, thin objects, and large motion. To address these issues, we introduce a patch-based cascaded pixel diffusion model for high resolution frame interpolation, HiFI, that excels in these scenarios while achieving competitive performance on standard benchmarks. Cascades, which generate a series of images from low to high resolution, can help significantly with large or complex motion that require both global context for a coarse solution and detailed context for high resolution output. However, contrary to prior work on cascaded diffusion models which perform diffusion on increasingly large resolutions, we use a single model that always performs diffusion at the same resolution and upsamples by processing patches of the inputs and the prior solution. At inference time, this drastically reduces memory usage and allows a single model, solving both frame interpolation (base model's task) and spatial up-sampling, saving training cost as well. HiFI excels at high-resolution images and complex repeated textures that require global context, achieving comparable or state-of-the-art performance on various benchmarks (Vimeo, Xiph, X-Test, and SEPE-8K). We further introduce a new dataset, LaMoR, that focuses on particularly challenging cases, and HiFI significantly outperforms other baselines. Please visit our project page for video results: https://hifi-diffusion.github.io |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_11838 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | High-Resolution Frame Interpolation with Patch-based Cascaded Diffusion Hur, Junhwa Herrmann, Charles Saxena, Saurabh Kontkanen, Janne Lai, Wei-Sheng Shih, Yichang Rubinstein, Michael Fleet, David J. Sun, Deqing Computer Vision and Pattern Recognition Despite the recent progress, existing frame interpolation methods still struggle with processing extremely high resolution input and handling challenging cases such as repetitive textures, thin objects, and large motion. To address these issues, we introduce a patch-based cascaded pixel diffusion model for high resolution frame interpolation, HiFI, that excels in these scenarios while achieving competitive performance on standard benchmarks. Cascades, which generate a series of images from low to high resolution, can help significantly with large or complex motion that require both global context for a coarse solution and detailed context for high resolution output. However, contrary to prior work on cascaded diffusion models which perform diffusion on increasingly large resolutions, we use a single model that always performs diffusion at the same resolution and upsamples by processing patches of the inputs and the prior solution. At inference time, this drastically reduces memory usage and allows a single model, solving both frame interpolation (base model's task) and spatial up-sampling, saving training cost as well. HiFI excels at high-resolution images and complex repeated textures that require global context, achieving comparable or state-of-the-art performance on various benchmarks (Vimeo, Xiph, X-Test, and SEPE-8K). We further introduce a new dataset, LaMoR, that focuses on particularly challenging cases, and HiFI significantly outperforms other baselines. Please visit our project page for video results: https://hifi-diffusion.github.io |
| title | High-Resolution Frame Interpolation with Patch-based Cascaded Diffusion |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.11838 |