Few-Shot-Based Modular Image-to-Video Adapter for Diffusion Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912796685369344 |
|---|---|
| author | Li, Zhenhao Yi, Shaohan Liu, Zheng Gao, Leonartinus Le, Minh Ngoc Ling, Ambrose Wang, Zhuoran Islam, Md Amirul Chi, Zhixiang Yu, Yuanhao |
| author_facet | Li, Zhenhao Yi, Shaohan Liu, Zheng Gao, Leonartinus Le, Minh Ngoc Ling, Ambrose Wang, Zhuoran Islam, Md Amirul Chi, Zhixiang Yu, Yuanhao |
| contents | Diffusion models (DMs) have recently achieved impressive photorealism in image and video generation. However, their application to image animation remains limited, even when trained on large-scale datasets. Two primary challenges contribute to this: the high dimensionality of video signals leads to a scarcity of training data, causing DMs to favor memorization over prompt compliance when generating motion; moreover, DMs struggle to generalize to novel motion patterns not present in the training set, and fine-tuning them to learn such patterns, especially using limited training data, is still under-explored. To address these limitations, we propose Modular Image-to-Video Adapter (MIVA), a lightweight sub-network attachable to a pre-trained DM, each designed to capture a single motion pattern and scalable via parallelization. MIVAs can be efficiently trained on approximately ten samples using a single consumer-grade GPU. At inference time, users can specify motion by selecting one or multiple MIVAs, eliminating the need for prompt engineering. Extensive experiments demonstrate that MIVA enables more precise motion control while maintaining, or even surpassing, the generation quality of models trained on significantly larger datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_20000 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Few-Shot-Based Modular Image-to-Video Adapter for Diffusion Models Li, Zhenhao Yi, Shaohan Liu, Zheng Gao, Leonartinus Le, Minh Ngoc Ling, Ambrose Wang, Zhuoran Islam, Md Amirul Chi, Zhixiang Yu, Yuanhao Computer Vision and Pattern Recognition Diffusion models (DMs) have recently achieved impressive photorealism in image and video generation. However, their application to image animation remains limited, even when trained on large-scale datasets. Two primary challenges contribute to this: the high dimensionality of video signals leads to a scarcity of training data, causing DMs to favor memorization over prompt compliance when generating motion; moreover, DMs struggle to generalize to novel motion patterns not present in the training set, and fine-tuning them to learn such patterns, especially using limited training data, is still under-explored. To address these limitations, we propose Modular Image-to-Video Adapter (MIVA), a lightweight sub-network attachable to a pre-trained DM, each designed to capture a single motion pattern and scalable via parallelization. MIVAs can be efficiently trained on approximately ten samples using a single consumer-grade GPU. At inference time, users can specify motion by selecting one or multiple MIVAs, eliminating the need for prompt engineering. Extensive experiments demonstrate that MIVA enables more precise motion control while maintaining, or even surpassing, the generation quality of models trained on significantly larger datasets. |
| title | Few-Shot-Based Modular Image-to-Video Adapter for Diffusion Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2512.20000 |