VASE: Object-Centric Appearance and Shape Manipulation of Real Videos
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917559039688704 |
|---|---|
| author | Peruzzo, Elia Goel, Vidit Xu, Dejia Xu, Xingqian Jiang, Yifan Wang, Zhangyang Shi, Humphrey Sebe, Nicu |
| author_facet | Peruzzo, Elia Goel, Vidit Xu, Dejia Xu, Xingqian Jiang, Yifan Wang, Zhangyang Shi, Humphrey Sebe, Nicu |
| contents | Recently, several works tackled the video editing task fostered by the success of large-scale text-to-image generative models. However, most of these methods holistically edit the frame using the text, exploiting the prior given by foundation diffusion models and focusing on improving the temporal consistency across frames. In this work, we introduce a framework that is object-centric and is designed to control both the object's appearance and, notably, to execute precise and explicit structural modifications on the object. We build our framework on a pre-trained image-conditioned diffusion model, integrate layers to handle the temporal dimension, and propose training strategies and architectural modifications to enable shape control. We evaluate our method on the image-driven video editing task showing similar performance to the state-of-the-art, and showcasing novel shape-editing capabilities. Further details, code and examples are available on our project page: https://helia95.github.io/vase-website/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_02473 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | VASE: Object-Centric Appearance and Shape Manipulation of Real Videos Peruzzo, Elia Goel, Vidit Xu, Dejia Xu, Xingqian Jiang, Yifan Wang, Zhangyang Shi, Humphrey Sebe, Nicu Computer Vision and Pattern Recognition Recently, several works tackled the video editing task fostered by the success of large-scale text-to-image generative models. However, most of these methods holistically edit the frame using the text, exploiting the prior given by foundation diffusion models and focusing on improving the temporal consistency across frames. In this work, we introduce a framework that is object-centric and is designed to control both the object's appearance and, notably, to execute precise and explicit structural modifications on the object. We build our framework on a pre-trained image-conditioned diffusion model, integrate layers to handle the temporal dimension, and propose training strategies and architectural modifications to enable shape control. We evaluate our method on the image-driven video editing task showing similar performance to the state-of-the-art, and showcasing novel shape-editing capabilities. Further details, code and examples are available on our project page: https://helia95.github.io/vase-website/ |
| title | VASE: Object-Centric Appearance and Shape Manipulation of Real Videos |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2401.02473 |