HARIVO: Harnessing Text-to-Image Models for Video Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910643861323776 |
|---|---|
| author | Kwon, Mingi Oh, Seoung Wug Zhou, Yang Liu, Difan Lee, Joon-Young Cai, Haoran Liu, Baqiao Liu, Feng Uh, Youngjung |
| author_facet | Kwon, Mingi Oh, Seoung Wug Zhou, Yang Liu, Difan Lee, Joon-Young Cai, Haoran Liu, Baqiao Liu, Feng Uh, Youngjung |
| contents | We present a method to create diffusion-based video models from pretrained Text-to-Image (T2I) models. Recently, AnimateDiff proposed freezing the T2I model while only training temporal layers. We advance this method by proposing a unique architecture, incorporating a mapping network and frame-wise tokens, tailored for video generation while maintaining the diversity and creativity of the original T2I model. Key innovations include novel loss functions for temporal smoothness and a mitigating gradient sampling technique, ensuring realistic and temporally consistent video generation despite limited public video data. We have successfully integrated video-specific inductive biases into the architecture and loss functions. Our method, built on the frozen StableDiffusion model, simplifies training processes and allows for seamless integration with off-the-shelf models like ControlNet and DreamBooth. project page: https://kwonminki.github.io/HARIVO |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_07763 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | HARIVO: Harnessing Text-to-Image Models for Video Generation Kwon, Mingi Oh, Seoung Wug Zhou, Yang Liu, Difan Lee, Joon-Young Cai, Haoran Liu, Baqiao Liu, Feng Uh, Youngjung Computer Vision and Pattern Recognition Artificial Intelligence We present a method to create diffusion-based video models from pretrained Text-to-Image (T2I) models. Recently, AnimateDiff proposed freezing the T2I model while only training temporal layers. We advance this method by proposing a unique architecture, incorporating a mapping network and frame-wise tokens, tailored for video generation while maintaining the diversity and creativity of the original T2I model. Key innovations include novel loss functions for temporal smoothness and a mitigating gradient sampling technique, ensuring realistic and temporally consistent video generation despite limited public video data. We have successfully integrated video-specific inductive biases into the architecture and loss functions. Our method, built on the frozen StableDiffusion model, simplifies training processes and allows for seamless integration with off-the-shelf models like ControlNet and DreamBooth. project page: https://kwonminki.github.io/HARIVO |
| title | HARIVO: Harnessing Text-to-Image Models for Video Generation |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2410.07763 |