Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2305.10474 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914728104689664 |
|---|---|
| author | Ge, Songwei Nah, Seungjun Liu, Guilin Poon, Tyler Tao, Andrew Catanzaro, Bryan Jacobs, David Huang, Jia-Bin Liu, Ming-Yu Balaji, Yogesh |
| author_facet | Ge, Songwei Nah, Seungjun Liu, Guilin Poon, Tyler Tao, Andrew Catanzaro, Bryan Jacobs, David Huang, Jia-Bin Liu, Ming-Yu Balaji, Yogesh |
| contents | Despite tremendous progress in generating high-quality images using diffusion models, synthesizing a sequence of animated frames that are both photorealistic and temporally coherent is still in its infancy. While off-the-shelf billion-scale datasets for image generation are available, collecting similar video data of the same scale is still challenging. Also, training a video diffusion model is computationally much more expensive than its image counterpart. In this work, we explore finetuning a pretrained image diffusion model with video data as a practical solution for the video synthesis task. We find that naively extending the image noise prior to video noise prior in video diffusion leads to sub-optimal performance. Our carefully designed video noise prior leads to substantially better performance. Extensive experimental validation shows that our model, Preserve Your Own Correlation (PYoCo), attains SOTA zero-shot text-to-video results on the UCF-101 and MSR-VTT benchmarks. It also achieves SOTA video generation quality on the small-scale UCF-101 benchmark with a $10\times$ smaller model using significantly less computation than the prior art. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_10474 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models Ge, Songwei Nah, Seungjun Liu, Guilin Poon, Tyler Tao, Andrew Catanzaro, Bryan Jacobs, David Huang, Jia-Bin Liu, Ming-Yu Balaji, Yogesh Computer Vision and Pattern Recognition Graphics Machine Learning Despite tremendous progress in generating high-quality images using diffusion models, synthesizing a sequence of animated frames that are both photorealistic and temporally coherent is still in its infancy. While off-the-shelf billion-scale datasets for image generation are available, collecting similar video data of the same scale is still challenging. Also, training a video diffusion model is computationally much more expensive than its image counterpart. In this work, we explore finetuning a pretrained image diffusion model with video data as a practical solution for the video synthesis task. We find that naively extending the image noise prior to video noise prior in video diffusion leads to sub-optimal performance. Our carefully designed video noise prior leads to substantially better performance. Extensive experimental validation shows that our model, Preserve Your Own Correlation (PYoCo), attains SOTA zero-shot text-to-video results on the UCF-101 and MSR-VTT benchmarks. It also achieves SOTA video generation quality on the small-scale UCF-101 benchmark with a $10\times$ smaller model using significantly less computation than the prior art. |
| title | Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models |
| topic | Computer Vision and Pattern Recognition Graphics Machine Learning |
| url | https://arxiv.org/abs/2305.10474 |