Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videos
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910285874331648 |
|---|---|
| author | Ma, Yue He, Yingqing Cun, Xiaodong Wang, Xintao Chen, Siran Shan, Ying Li, Xiu Chen, Qifeng |
| author_facet | Ma, Yue He, Yingqing Cun, Xiaodong Wang, Xintao Chen, Siran Shan, Ying Li, Xiu Chen, Qifeng |
| contents | Generating text-editable and pose-controllable character videos have an imperious demand in creating various digital human. Nevertheless, this task has been restricted by the absence of a comprehensive dataset featuring paired video-pose captions and the generative prior models for videos. In this work, we design a novel two-stage training scheme that can utilize easily obtained datasets (i.e.,image pose pair and pose-free video) and the pre-trained text-to-image (T2I) model to obtain the pose-controllable character videos. Specifically, in the first stage, only the keypoint-image pairs are used only for a controllable text-to-image generation. We learn a zero-initialized convolutional encoder to encode the pose information. In the second stage, we finetune the motion of the above network via a pose-free video dataset by adding the learnable temporal self-attention and reformed cross-frame self-attention blocks. Powered by our new designs, our method successfully generates continuously pose-controllable character videos while keeps the editing and concept composition ability of the pre-trained T2I model. The code and models will be made publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2304_01186 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videos Ma, Yue He, Yingqing Cun, Xiaodong Wang, Xintao Chen, Siran Shan, Ying Li, Xiu Chen, Qifeng Computer Vision and Pattern Recognition Generating text-editable and pose-controllable character videos have an imperious demand in creating various digital human. Nevertheless, this task has been restricted by the absence of a comprehensive dataset featuring paired video-pose captions and the generative prior models for videos. In this work, we design a novel two-stage training scheme that can utilize easily obtained datasets (i.e.,image pose pair and pose-free video) and the pre-trained text-to-image (T2I) model to obtain the pose-controllable character videos. Specifically, in the first stage, only the keypoint-image pairs are used only for a controllable text-to-image generation. We learn a zero-initialized convolutional encoder to encode the pose information. In the second stage, we finetune the motion of the above network via a pose-free video dataset by adding the learnable temporal self-attention and reformed cross-frame self-attention blocks. Powered by our new designs, our method successfully generates continuously pose-controllable character videos while keeps the editing and concept composition ability of the pre-trained T2I model. The code and models will be made publicly available. |
| title | Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videos |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2304.01186 |