Text Prompting for Multi-Concept Video Customization by Autoregressive Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929354262446080 |
|---|---|
| author | Kothandaraman, Divya Sohn, Kihyuk Villegas, Ruben Voigtlaender, Paul Manocha, Dinesh Babaeizadeh, Mohammad |
| author_facet | Kothandaraman, Divya Sohn, Kihyuk Villegas, Ruben Voigtlaender, Paul Manocha, Dinesh Babaeizadeh, Mohammad |
| contents | We present a method for multi-concept customization of pretrained text-to-video (T2V) models. Intuitively, the multi-concept customized video can be derived from the (non-linear) intersection of the video manifolds of the individual concepts, which is not straightforward to find. We hypothesize that sequential and controlled walking towards the intersection of the video manifolds, directed by text prompting, leads to the solution. To do so, we generate the various concepts and their corresponding interactions, sequentially, in an autoregressive manner. Our method can generate videos of multiple custom concepts (subjects, action and background) such as a teddy bear running towards a brown teapot, a dog playing violin and a teddy bear swimming in the ocean. We quantitatively evaluate our method using videoCLIP and DINO scores, in addition to human evaluation. Videos for results presented in this paper can be found at https://github.com/divyakraman/MultiConceptVideo2024. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_13951 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Text Prompting for Multi-Concept Video Customization by Autoregressive Generation Kothandaraman, Divya Sohn, Kihyuk Villegas, Ruben Voigtlaender, Paul Manocha, Dinesh Babaeizadeh, Mohammad Computer Vision and Pattern Recognition We present a method for multi-concept customization of pretrained text-to-video (T2V) models. Intuitively, the multi-concept customized video can be derived from the (non-linear) intersection of the video manifolds of the individual concepts, which is not straightforward to find. We hypothesize that sequential and controlled walking towards the intersection of the video manifolds, directed by text prompting, leads to the solution. To do so, we generate the various concepts and their corresponding interactions, sequentially, in an autoregressive manner. Our method can generate videos of multiple custom concepts (subjects, action and background) such as a teddy bear running towards a brown teapot, a dog playing violin and a teddy bear swimming in the ocean. We quantitatively evaluate our method using videoCLIP and DINO scores, in addition to human evaluation. Videos for results presented in this paper can be found at https://github.com/divyakraman/MultiConceptVideo2024. |
| title | Text Prompting for Multi-Concept Video Customization by Autoregressive Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2405.13951 |