I2V-Adapter: A General Image-to-Video Adapter for Diffusion Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910503360528384 |
|---|---|
| author | Guo, Xun Zheng, Mingwu Hou, Liang Gao, Yuan Deng, Yufan Wan, Pengfei Zhang, Di Liu, Yufan Hu, Weiming Zha, Zhengjun Huang, Haibin Ma, Chongyang |
| author_facet | Guo, Xun Zheng, Mingwu Hou, Liang Gao, Yuan Deng, Yufan Wan, Pengfei Zhang, Di Liu, Yufan Hu, Weiming Zha, Zhengjun Huang, Haibin Ma, Chongyang |
| contents | Text-guided image-to-video (I2V) generation aims to generate a coherent video that preserves the identity of the input image and semantically aligns with the input prompt. Existing methods typically augment pretrained text-to-video (T2V) models by either concatenating the image with noised video frames channel-wise before being fed into the model or injecting the image embedding produced by pretrained image encoders in cross-attention modules. However, the former approach often necessitates altering the fundamental weights of pretrained T2V models, thus restricting the model's compatibility within the open-source communities and disrupting the model's prior knowledge. Meanwhile, the latter typically fails to preserve the identity of the input image. We present I2V-Adapter to overcome such limitations. I2V-Adapter adeptly propagates the unnoised input image to subsequent noised frames through a cross-frame attention mechanism, maintaining the identity of the input image without any changes to the pretrained T2V model. Notably, I2V-Adapter only introduces a few trainable parameters, significantly alleviating the training cost and also ensures compatibility with existing community-driven personalized models and control tools. Moreover, we propose a novel Frame Similarity Prior to balance the motion amplitude and the stability of generated videos through two adjustable control coefficients. Our experimental results demonstrate that I2V-Adapter is capable of producing high-quality videos. This performance, coupled with its agility and adaptability, represents a substantial advancement in the field of I2V, particularly for personalized and controllable applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_16693 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | I2V-Adapter: A General Image-to-Video Adapter for Diffusion Models Guo, Xun Zheng, Mingwu Hou, Liang Gao, Yuan Deng, Yufan Wan, Pengfei Zhang, Di Liu, Yufan Hu, Weiming Zha, Zhengjun Huang, Haibin Ma, Chongyang Computer Vision and Pattern Recognition Text-guided image-to-video (I2V) generation aims to generate a coherent video that preserves the identity of the input image and semantically aligns with the input prompt. Existing methods typically augment pretrained text-to-video (T2V) models by either concatenating the image with noised video frames channel-wise before being fed into the model or injecting the image embedding produced by pretrained image encoders in cross-attention modules. However, the former approach often necessitates altering the fundamental weights of pretrained T2V models, thus restricting the model's compatibility within the open-source communities and disrupting the model's prior knowledge. Meanwhile, the latter typically fails to preserve the identity of the input image. We present I2V-Adapter to overcome such limitations. I2V-Adapter adeptly propagates the unnoised input image to subsequent noised frames through a cross-frame attention mechanism, maintaining the identity of the input image without any changes to the pretrained T2V model. Notably, I2V-Adapter only introduces a few trainable parameters, significantly alleviating the training cost and also ensures compatibility with existing community-driven personalized models and control tools. Moreover, we propose a novel Frame Similarity Prior to balance the motion amplitude and the stability of generated videos through two adjustable control coefficients. Our experimental results demonstrate that I2V-Adapter is capable of producing high-quality videos. This performance, coupled with its agility and adaptability, represents a substantial advancement in the field of I2V, particularly for personalized and controllable applications. |
| title | I2V-Adapter: A General Image-to-Video Adapter for Diffusion Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2312.16693 |