Motion Prompting: Controlling Video Generation with Motion Trajectories
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910897164779520 |
|---|---|
| author | Geng, Daniel Herrmann, Charles Hur, Junhwa Cole, Forrester Zhang, Serena Pfaff, Tobias Lopez-Guevara, Tatiana Doersch, Carl Aytar, Yusuf Rubinstein, Michael Sun, Chen Wang, Oliver Owens, Andrew Sun, Deqing |
| author_facet | Geng, Daniel Herrmann, Charles Hur, Junhwa Cole, Forrester Zhang, Serena Pfaff, Tobias Lopez-Guevara, Tatiana Doersch, Carl Aytar, Yusuf Rubinstein, Michael Sun, Chen Wang, Oliver Owens, Andrew Sun, Deqing |
| contents | Motion control is crucial for generating expressive and compelling video content; however, most existing video generation models rely mainly on text prompts for control, which struggle to capture the nuances of dynamic actions and temporal compositions. To this end, we train a video generation model conditioned on spatio-temporally sparse or dense motion trajectories. In contrast to prior motion conditioning work, this flexible representation can encode any number of trajectories, object-specific or global scene motion, and temporally sparse motion; due to its flexibility we refer to this conditioning as motion prompts. While users may directly specify sparse trajectories, we also show how to translate high-level user requests into detailed, semi-dense motion prompts, a process we term motion prompt expansion. We demonstrate the versatility of our approach through various applications, including camera and object motion control, "interacting" with an image, motion transfer, and image editing. Our results showcase emergent behaviors, such as realistic physics, suggesting the potential of motion prompts for probing video models and interacting with future generative world models. Finally, we evaluate quantitatively, conduct a human study, and demonstrate strong performance. Video results are available on our webpage: https://motion-prompting.github.io/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_02700 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Motion Prompting: Controlling Video Generation with Motion Trajectories Geng, Daniel Herrmann, Charles Hur, Junhwa Cole, Forrester Zhang, Serena Pfaff, Tobias Lopez-Guevara, Tatiana Doersch, Carl Aytar, Yusuf Rubinstein, Michael Sun, Chen Wang, Oliver Owens, Andrew Sun, Deqing Computer Vision and Pattern Recognition Motion control is crucial for generating expressive and compelling video content; however, most existing video generation models rely mainly on text prompts for control, which struggle to capture the nuances of dynamic actions and temporal compositions. To this end, we train a video generation model conditioned on spatio-temporally sparse or dense motion trajectories. In contrast to prior motion conditioning work, this flexible representation can encode any number of trajectories, object-specific or global scene motion, and temporally sparse motion; due to its flexibility we refer to this conditioning as motion prompts. While users may directly specify sparse trajectories, we also show how to translate high-level user requests into detailed, semi-dense motion prompts, a process we term motion prompt expansion. We demonstrate the versatility of our approach through various applications, including camera and object motion control, "interacting" with an image, motion transfer, and image editing. Our results showcase emergent behaviors, such as realistic physics, suggesting the potential of motion prompts for probing video models and interacting with future generative world models. Finally, we evaluate quantitatively, conduct a human study, and demonstrate strong performance. Video results are available on our webpage: https://motion-prompting.github.io/ |
| title | Motion Prompting: Controlling Video Generation with Motion Trajectories |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.02700 |