Lumiere: A Space-Time Diffusion Model for Video Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911770709327872 |
|---|---|
| author | Bar-Tal, Omer Chefer, Hila Tov, Omer Herrmann, Charles Paiss, Roni Zada, Shiran Ephrat, Ariel Hur, Junhwa Liu, Guanghui Raj, Amit Li, Yuanzhen Rubinstein, Michael Michaeli, Tomer Wang, Oliver Sun, Deqing Dekel, Tali Mosseri, Inbar |
| author_facet | Bar-Tal, Omer Chefer, Hila Tov, Omer Herrmann, Charles Paiss, Roni Zada, Shiran Ephrat, Ariel Hur, Junhwa Liu, Guanghui Raj, Amit Li, Yuanzhen Rubinstein, Michael Michaeli, Tomer Wang, Oliver Sun, Deqing Dekel, Tali Mosseri, Inbar |
| contents | We introduce Lumiere -- a text-to-video diffusion model designed for synthesizing videos that portray realistic, diverse and coherent motion -- a pivotal challenge in video synthesis. To this end, we introduce a Space-Time U-Net architecture that generates the entire temporal duration of the video at once, through a single pass in the model. This is in contrast to existing video models which synthesize distant keyframes followed by temporal super-resolution -- an approach that inherently makes global temporal consistency difficult to achieve. By deploying both spatial and (importantly) temporal down- and up-sampling and leveraging a pre-trained text-to-image diffusion model, our model learns to directly generate a full-frame-rate, low-resolution video by processing it in multiple space-time scales. We demonstrate state-of-the-art text-to-video generation results, and show that our design easily facilitates a wide range of content creation tasks and video editing applications, including image-to-video, video inpainting, and stylized generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_12945 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Lumiere: A Space-Time Diffusion Model for Video Generation Bar-Tal, Omer Chefer, Hila Tov, Omer Herrmann, Charles Paiss, Roni Zada, Shiran Ephrat, Ariel Hur, Junhwa Liu, Guanghui Raj, Amit Li, Yuanzhen Rubinstein, Michael Michaeli, Tomer Wang, Oliver Sun, Deqing Dekel, Tali Mosseri, Inbar Computer Vision and Pattern Recognition We introduce Lumiere -- a text-to-video diffusion model designed for synthesizing videos that portray realistic, diverse and coherent motion -- a pivotal challenge in video synthesis. To this end, we introduce a Space-Time U-Net architecture that generates the entire temporal duration of the video at once, through a single pass in the model. This is in contrast to existing video models which synthesize distant keyframes followed by temporal super-resolution -- an approach that inherently makes global temporal consistency difficult to achieve. By deploying both spatial and (importantly) temporal down- and up-sampling and leveraging a pre-trained text-to-image diffusion model, our model learns to directly generate a full-frame-rate, low-resolution video by processing it in multiple space-time scales. We demonstrate state-of-the-art text-to-video generation results, and show that our design easily facilitates a wide range of content creation tasks and video editing applications, including image-to-video, video inpainting, and stylized generation. |
| title | Lumiere: A Space-Time Diffusion Model for Video Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2401.12945 |