Mojito: Motion Trajectory and Intensity Control for Video Generation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915137935376384 |
|---|---|
| author | He, Xuehai Wang, Shuohang Yang, Jianwei Wu, Xiaoxia Wang, Yiping Wang, Kuan Zhan, Zheng Ruwase, Olatunji Shen, Yelong Wang, Xin Eric |
| author_facet | He, Xuehai Wang, Shuohang Yang, Jianwei Wu, Xiaoxia Wang, Yiping Wang, Kuan Zhan, Zheng Ruwase, Olatunji Shen, Yelong Wang, Xin Eric |
| contents | Recent advancements in diffusion models have shown great promise in producing high-quality video content. However, efficiently training video diffusion models capable of integrating directional guidance and controllable motion intensity remains a challenging and under-explored area. To tackle these challenges, this paper introduces Mojito, a diffusion model that incorporates both motion trajectory and intensity control for text-to-video generation. Specifically, Mojito features a Directional Motion Control (DMC) module that leverages cross-attention to efficiently direct the generated object's motion without training, alongside a Motion Intensity Modulator (MIM) that uses optical flow maps generated from videos to guide varying levels of motion intensity. Extensive experiments demonstrate Mojito's effectiveness in achieving precise trajectory and intensity control with high computational efficiency, generating motion patterns that closely match specified directions and intensities, providing realistic dynamics that align well with natural motion in real-world scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_08948 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Mojito: Motion Trajectory and Intensity Control for Video Generation He, Xuehai Wang, Shuohang Yang, Jianwei Wu, Xiaoxia Wang, Yiping Wang, Kuan Zhan, Zheng Ruwase, Olatunji Shen, Yelong Wang, Xin Eric Computer Vision and Pattern Recognition Computation and Language Recent advancements in diffusion models have shown great promise in producing high-quality video content. However, efficiently training video diffusion models capable of integrating directional guidance and controllable motion intensity remains a challenging and under-explored area. To tackle these challenges, this paper introduces Mojito, a diffusion model that incorporates both motion trajectory and intensity control for text-to-video generation. Specifically, Mojito features a Directional Motion Control (DMC) module that leverages cross-attention to efficiently direct the generated object's motion without training, alongside a Motion Intensity Modulator (MIM) that uses optical flow maps generated from videos to guide varying levels of motion intensity. Extensive experiments demonstrate Mojito's effectiveness in achieving precise trajectory and intensity control with high computational efficiency, generating motion patterns that closely match specified directions and intensities, providing realistic dynamics that align well with natural motion in real-world scenarios. |
| title | Mojito: Motion Trajectory and Intensity Control for Video Generation |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2412.08948 |