Video Creation by Demonstration
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913609714499584 |
|---|---|
| author | Sun, Yihong Zhou, Hao Yuan, Liangzhe Sun, Jennifer J. Li, Yandong Jia, Xuhui Adam, Hartwig Hariharan, Bharath Zhao, Long Liu, Ting |
| author_facet | Sun, Yihong Zhou, Hao Yuan, Liangzhe Sun, Jennifer J. Li, Yandong Jia, Xuhui Adam, Hartwig Hariharan, Bharath Zhao, Long Liu, Ting |
| contents | We explore a novel video creation experience, namely Video Creation by Demonstration. Given a demonstration video and a context image from a different scene, we generate a physically plausible video that continues naturally from the context image and carries out the action concepts from the demonstration. To enable this capability, we present $δ$-Diffusion, a self-supervised training approach that learns from unlabeled videos by conditional future frame prediction. Unlike most existing video generation controls that are based on explicit signals, we adopts the form of implicit latent control for maximal flexibility and expressiveness required by general videos. By leveraging a video foundation model with an appearance bottleneck design on top, we extract action latents from demonstration videos for conditioning the generation process with minimal appearance leakage. Empirically, $δ$-Diffusion outperforms related baselines in terms of both human preference and large-scale machine evaluations, and demonstrates potentials towards interactive world simulation. Sampled video generation results are available at https://delta-diffusion.github.io/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_09551 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Video Creation by Demonstration Sun, Yihong Zhou, Hao Yuan, Liangzhe Sun, Jennifer J. Li, Yandong Jia, Xuhui Adam, Hartwig Hariharan, Bharath Zhao, Long Liu, Ting Computer Vision and Pattern Recognition We explore a novel video creation experience, namely Video Creation by Demonstration. Given a demonstration video and a context image from a different scene, we generate a physically plausible video that continues naturally from the context image and carries out the action concepts from the demonstration. To enable this capability, we present $δ$-Diffusion, a self-supervised training approach that learns from unlabeled videos by conditional future frame prediction. Unlike most existing video generation controls that are based on explicit signals, we adopts the form of implicit latent control for maximal flexibility and expressiveness required by general videos. By leveraging a video foundation model with an appearance bottleneck design on top, we extract action latents from demonstration videos for conditioning the generation process with minimal appearance leakage. Empirically, $δ$-Diffusion outperforms related baselines in terms of both human preference and large-scale machine evaluations, and demonstrates potentials towards interactive world simulation. Sampled video generation results are available at https://delta-diffusion.github.io/. |
| title | Video Creation by Demonstration |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.09551 |