SF-V: Single Forward Video Generation Model
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910666102669312 |
|---|---|
| author | Zhang, Zhixing Li, Yanyu Wu, Yushu Xu, Yanwu Kag, Anil Skorokhodov, Ivan Menapace, Willi Siarohin, Aliaksandr Cao, Junli Metaxas, Dimitris Tulyakov, Sergey Ren, Jian |
| author_facet | Zhang, Zhixing Li, Yanyu Wu, Yushu Xu, Yanwu Kag, Anil Skorokhodov, Ivan Menapace, Willi Siarohin, Aliaksandr Cao, Junli Metaxas, Dimitris Tulyakov, Sergey Ren, Jian |
| contents | Diffusion-based video generation models have demonstrated remarkable success in obtaining high-fidelity videos through the iterative denoising process. However, these models require multiple denoising steps during sampling, resulting in high computational costs. In this work, we propose a novel approach to obtain single-step video generation models by leveraging adversarial training to fine-tune pre-trained video diffusion models. We show that, through the adversarial training, the multi-steps video diffusion model, i.e., Stable Video Diffusion (SVD), can be trained to perform single forward pass to synthesize high-quality videos, capturing both temporal and spatial dependencies in the video data. Extensive experiments demonstrate that our method achieves competitive generation quality of synthesized videos with significantly reduced computational overhead for the denoising process (i.e., around $23\times$ speedup compared with SVD and $6\times$ speedup compared with existing works, with even better generation quality), paving the way for real-time video synthesis and editing. More visualization results are made publicly available at https://snap-research.github.io/SF-V. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_04324 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SF-V: Single Forward Video Generation Model Zhang, Zhixing Li, Yanyu Wu, Yushu Xu, Yanwu Kag, Anil Skorokhodov, Ivan Menapace, Willi Siarohin, Aliaksandr Cao, Junli Metaxas, Dimitris Tulyakov, Sergey Ren, Jian Computer Vision and Pattern Recognition Image and Video Processing Diffusion-based video generation models have demonstrated remarkable success in obtaining high-fidelity videos through the iterative denoising process. However, these models require multiple denoising steps during sampling, resulting in high computational costs. In this work, we propose a novel approach to obtain single-step video generation models by leveraging adversarial training to fine-tune pre-trained video diffusion models. We show that, through the adversarial training, the multi-steps video diffusion model, i.e., Stable Video Diffusion (SVD), can be trained to perform single forward pass to synthesize high-quality videos, capturing both temporal and spatial dependencies in the video data. Extensive experiments demonstrate that our method achieves competitive generation quality of synthesized videos with significantly reduced computational overhead for the denoising process (i.e., around $23\times$ speedup compared with SVD and $6\times$ speedup compared with existing works, with even better generation quality), paving the way for real-time video synthesis and editing. More visualization results are made publicly available at https://snap-research.github.io/SF-V. |
| title | SF-V: Single Forward Video Generation Model |
| topic | Computer Vision and Pattern Recognition Image and Video Processing |
| url | https://arxiv.org/abs/2406.04324 |