Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2408.11475 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929659104460800 |
|---|---|
| author | Zhou, Haitao Wang, Chuang Nie, Rui Liu, Jinlin Yu, Dongdong Yu, Qian Wang, Changhu |
| author_facet | Zhou, Haitao Wang, Chuang Nie, Rui Liu, Jinlin Yu, Dongdong Yu, Qian Wang, Changhu |
| contents | Recent years have seen substantial progress in diffusion-based controllable video generation. However, achieving precise control in complex scenarios, including fine-grained object parts, sophisticated motion trajectories, and coherent background movement, remains a challenge. In this paper, we introduce TrackGo, a novel approach that leverages free-form masks and arrows for conditional video generation. This method offers users with a flexible and precise mechanism for manipulating video content. We also propose the TrackAdapter for control implementation, an efficient and lightweight adapter designed to be seamlessly integrated into the temporal self-attention layers of a pretrained video generation model. This design leverages our observation that the attention map of these layers can accurately activate regions corresponding to motion in videos. Our experimental results demonstrate that our new approach, enhanced by the TrackAdapter, achieves state-of-the-art performance on key metrics such as FVD, FID, and ObjMC scores. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_11475 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TrackGo: A Flexible and Efficient Method for Controllable Video Generation Zhou, Haitao Wang, Chuang Nie, Rui Liu, Jinlin Yu, Dongdong Yu, Qian Wang, Changhu Computer Vision and Pattern Recognition Recent years have seen substantial progress in diffusion-based controllable video generation. However, achieving precise control in complex scenarios, including fine-grained object parts, sophisticated motion trajectories, and coherent background movement, remains a challenge. In this paper, we introduce TrackGo, a novel approach that leverages free-form masks and arrows for conditional video generation. This method offers users with a flexible and precise mechanism for manipulating video content. We also propose the TrackAdapter for control implementation, an efficient and lightweight adapter designed to be seamlessly integrated into the temporal self-attention layers of a pretrained video generation model. This design leverages our observation that the attention map of these layers can accurately activate regions corresponding to motion in videos. Our experimental results demonstrate that our new approach, enhanced by the TrackAdapter, achieves state-of-the-art performance on key metrics such as FVD, FID, and ObjMC scores. |
| title | TrackGo: A Flexible and Efficient Method for Controllable Video Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2408.11475 |