Generative Inbetweening: Adapting Image-to-Video Models for Keyframe Interpolation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909488819208192 |
|---|---|
| author | Wang, Xiaojuan Zhou, Boyang Curless, Brian Kemelmacher-Shlizerman, Ira Holynski, Aleksander Seitz, Steven M. |
| author_facet | Wang, Xiaojuan Zhou, Boyang Curless, Brian Kemelmacher-Shlizerman, Ira Holynski, Aleksander Seitz, Steven M. |
| contents | We present a method for generating video sequences with coherent motion between a pair of input key frames. We adapt a pretrained large-scale image-to-video diffusion model (originally trained to generate videos moving forward in time from a single input image) for key frame interpolation, i.e., to produce a video in between two input frames. We accomplish this adaptation through a lightweight fine-tuning technique that produces a version of the model that instead predicts videos moving backwards in time from a single input image. This model (along with the original forward-moving model) is subsequently used in a dual-directional diffusion sampling process that combines the overlapping model estimates starting from each of the two keyframes. Our experiments show that our method outperforms both existing diffusion-based methods and traditional frame interpolation techniques. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_15239 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Generative Inbetweening: Adapting Image-to-Video Models for Keyframe Interpolation Wang, Xiaojuan Zhou, Boyang Curless, Brian Kemelmacher-Shlizerman, Ira Holynski, Aleksander Seitz, Steven M. Computer Vision and Pattern Recognition We present a method for generating video sequences with coherent motion between a pair of input key frames. We adapt a pretrained large-scale image-to-video diffusion model (originally trained to generate videos moving forward in time from a single input image) for key frame interpolation, i.e., to produce a video in between two input frames. We accomplish this adaptation through a lightweight fine-tuning technique that produces a version of the model that instead predicts videos moving backwards in time from a single input image. This model (along with the original forward-moving model) is subsequently used in a dual-directional diffusion sampling process that combines the overlapping model estimates starting from each of the two keyframes. Our experiments show that our method outperforms both existing diffusion-based methods and traditional frame interpolation techniques. |
| title | Generative Inbetweening: Adapting Image-to-Video Models for Keyframe Interpolation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2408.15239 |