Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.13552 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915440671850496 |
|---|---|
| author | Tian, Rui Dai, Qi Bao, Jianmin Qiu, Kai Yang, Yifan Luo, Chong Wu, Zuxuan Jiang, Yu-Gang |
| author_facet | Tian, Rui Dai, Qi Bao, Jianmin Qiu, Kai Yang, Yifan Luo, Chong Wu, Zuxuan Jiang, Yu-Gang |
| contents | Commercial video generation models have exhibited realistic, high-fidelity results but are still restricted to limited access. One crucial obstacle for large-scale applications is the expensive training and inference cost. In this paper, we argue that videos contain significantly more redundant information than images, allowing them to be encoded with very few motion latents. Towards this goal, we design an image-conditioned VAE that projects videos into extremely compressed latent space and decode them based on content images. This magic Reducio charm enables 64x reduction of latents compared to a common 2D VAE, without sacrificing the quality. Building upon Reducio-VAE, we can train diffusion models for high-resolution video generation efficiently. Specifically, we adopt a two-stage generation paradigm, first generating a condition image via text-to-image generation, followed by text-image-to-video generation with the proposed Reducio-DiT. Extensive experiments show that our model achieves strong performance in evaluation. More importantly, our method significantly boosts the training and inference efficiency of video LDMs. Reducio-DiT is trained in just 3.2K A100 GPU hours in total and can generate a 16-frame 1024$\times$1024 video clip within 15.5 seconds on a single A100 GPU. Code released at https://github.com/microsoft/Reducio-VAE . |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_13552 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | REDUCIO! Generating 1K Video within 16 Seconds using Extremely Compressed Motion Latents Tian, Rui Dai, Qi Bao, Jianmin Qiu, Kai Yang, Yifan Luo, Chong Wu, Zuxuan Jiang, Yu-Gang Computer Vision and Pattern Recognition Commercial video generation models have exhibited realistic, high-fidelity results but are still restricted to limited access. One crucial obstacle for large-scale applications is the expensive training and inference cost. In this paper, we argue that videos contain significantly more redundant information than images, allowing them to be encoded with very few motion latents. Towards this goal, we design an image-conditioned VAE that projects videos into extremely compressed latent space and decode them based on content images. This magic Reducio charm enables 64x reduction of latents compared to a common 2D VAE, without sacrificing the quality. Building upon Reducio-VAE, we can train diffusion models for high-resolution video generation efficiently. Specifically, we adopt a two-stage generation paradigm, first generating a condition image via text-to-image generation, followed by text-image-to-video generation with the proposed Reducio-DiT. Extensive experiments show that our model achieves strong performance in evaluation. More importantly, our method significantly boosts the training and inference efficiency of video LDMs. Reducio-DiT is trained in just 3.2K A100 GPU hours in total and can generate a 16-frame 1024$\times$1024 video clip within 15.5 seconds on a single A100 GPU. Code released at https://github.com/microsoft/Reducio-VAE . |
| title | REDUCIO! Generating 1K Video within 16 Seconds using Extremely Compressed Motion Latents |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2411.13552 |