I2VControl: Disentangled and Unified Video Motion Synthesis Control
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918107486879744 |
|---|---|
| author | Feng, Wanquan Qi, Tianhao Liu, Jiawei Sun, Mingzhen Tu, Pengqi Ma, Tianxiang Dai, Fei Zhao, Songtao Zhou, Siyu He, Qian |
| author_facet | Feng, Wanquan Qi, Tianhao Liu, Jiawei Sun, Mingzhen Tu, Pengqi Ma, Tianxiang Dai, Fei Zhao, Songtao Zhou, Siyu He, Qian |
| contents | Motion controllability is crucial in video synthesis. However, most previous methods are limited to single control types, and combining them often results in logical conflicts. In this paper, we propose a disentangled and unified framework, namely I2VControl, to overcome the logical conflicts. We rethink camera control, object dragging, and motion brush, reformulating all tasks into a consistent representation based on point trajectories, each managed by a dedicated formulation. Accordingly, we propose a spatial partitioning strategy, where each unit is assigned to a concomitant control category, enabling diverse control types to be dynamically orchestrated within a single synthesis pipeline without conflicts. Furthermore, we design an adapter structure that functions as a plug-in for pre-trained models and is agnostic to specific model architectures. We conduct extensive experiments, achieving excellent performance on various control tasks, and our method further facilitates user-driven creative combinations, enhancing innovation and creativity. Project page: https://wanquanf.github.io/I2VControl . |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_17765 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | I2VControl: Disentangled and Unified Video Motion Synthesis Control Feng, Wanquan Qi, Tianhao Liu, Jiawei Sun, Mingzhen Tu, Pengqi Ma, Tianxiang Dai, Fei Zhao, Songtao Zhou, Siyu He, Qian Computer Vision and Pattern Recognition Motion controllability is crucial in video synthesis. However, most previous methods are limited to single control types, and combining them often results in logical conflicts. In this paper, we propose a disentangled and unified framework, namely I2VControl, to overcome the logical conflicts. We rethink camera control, object dragging, and motion brush, reformulating all tasks into a consistent representation based on point trajectories, each managed by a dedicated formulation. Accordingly, we propose a spatial partitioning strategy, where each unit is assigned to a concomitant control category, enabling diverse control types to be dynamically orchestrated within a single synthesis pipeline without conflicts. Furthermore, we design an adapter structure that functions as a plug-in for pre-trained models and is agnostic to specific model architectures. We conduct extensive experiments, achieving excellent performance on various control tasks, and our method further facilitates user-driven creative combinations, enhancing innovation and creativity. Project page: https://wanquanf.github.io/I2VControl . |
| title | I2VControl: Disentangled and Unified Video Motion Synthesis Control |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2411.17765 |