InteractiveVideo: User-Centric Controllable Video Generation with Synergistic Multimodal Instructions
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909093605670912 |
|---|---|
| author | Zhang, Yiyuan Kang, Yuhao Zhang, Zhixin Ding, Xiaohan Zhao, Sanyuan Yue, Xiangyu |
| author_facet | Zhang, Yiyuan Kang, Yuhao Zhang, Zhixin Ding, Xiaohan Zhao, Sanyuan Yue, Xiangyu |
| contents | We introduce $\textit{InteractiveVideo}$, a user-centric framework for video generation. Different from traditional generative approaches that operate based on user-provided images or text, our framework is designed for dynamic interaction, allowing users to instruct the generative model through various intuitive mechanisms during the whole generation process, e.g. text and image prompts, painting, drag-and-drop, etc. We propose a Synergistic Multimodal Instruction mechanism, designed to seamlessly integrate users' multimodal instructions into generative models, thus facilitating a cooperative and responsive interaction between user inputs and the generative process. This approach enables iterative and fine-grained refinement of the generation result through precise and effective user instructions. With $\textit{InteractiveVideo}$, users are given the flexibility to meticulously tailor key aspects of a video. They can paint the reference image, edit semantics, and adjust video motions until their requirements are fully met. Code, models, and demo are available at https://github.com/invictus717/InteractiveVideo |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_03040 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | InteractiveVideo: User-Centric Controllable Video Generation with Synergistic Multimodal Instructions Zhang, Yiyuan Kang, Yuhao Zhang, Zhixin Ding, Xiaohan Zhao, Sanyuan Yue, Xiangyu Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Multimedia We introduce $\textit{InteractiveVideo}$, a user-centric framework for video generation. Different from traditional generative approaches that operate based on user-provided images or text, our framework is designed for dynamic interaction, allowing users to instruct the generative model through various intuitive mechanisms during the whole generation process, e.g. text and image prompts, painting, drag-and-drop, etc. We propose a Synergistic Multimodal Instruction mechanism, designed to seamlessly integrate users' multimodal instructions into generative models, thus facilitating a cooperative and responsive interaction between user inputs and the generative process. This approach enables iterative and fine-grained refinement of the generation result through precise and effective user instructions. With $\textit{InteractiveVideo}$, users are given the flexibility to meticulously tailor key aspects of a video. They can paint the reference image, edit semantics, and adjust video motions until their requirements are fully met. Code, models, and demo are available at https://github.com/invictus717/InteractiveVideo |
| title | InteractiveVideo: User-Centric Controllable Video Generation with Synergistic Multimodal Instructions |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Multimedia |
| url | https://arxiv.org/abs/2402.03040 |