VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913752556765184 |
|---|---|
| author | Bahmani, Sherwin Skorokhodov, Ivan Siarohin, Aliaksandr Menapace, Willi Qian, Guocheng Vasilkovsky, Michael Lee, Hsin-Ying Wang, Chaoyang Zou, Jiaxu Tagliasacchi, Andrea Lindell, David B. Tulyakov, Sergey |
| author_facet | Bahmani, Sherwin Skorokhodov, Ivan Siarohin, Aliaksandr Menapace, Willi Qian, Guocheng Vasilkovsky, Michael Lee, Hsin-Ying Wang, Chaoyang Zou, Jiaxu Tagliasacchi, Andrea Lindell, David B. Tulyakov, Sergey |
| contents | Modern text-to-video synthesis models demonstrate coherent, photorealistic generation of complex videos from a text description. However, most existing models lack fine-grained control over camera movement, which is critical for downstream applications related to content creation, visual effects, and 3D vision. Recently, new methods demonstrate the ability to generate videos with controllable camera poses these techniques leverage pre-trained U-Net-based diffusion models that explicitly disentangle spatial and temporal generation. Still, no existing approach enables camera control for new, transformer-based video diffusion models that process spatial and temporal information jointly. Here, we propose to tame video transformers for 3D camera control using a ControlNet-like conditioning mechanism that incorporates spatiotemporal camera embeddings based on Plücker coordinates. The approach demonstrates state-of-the-art performance for controllable video generation after fine-tuning on the RealEstate10K dataset. To the best of our knowledge, our work is the first to enable camera control for transformer-based video diffusion models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_12781 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control Bahmani, Sherwin Skorokhodov, Ivan Siarohin, Aliaksandr Menapace, Willi Qian, Guocheng Vasilkovsky, Michael Lee, Hsin-Ying Wang, Chaoyang Zou, Jiaxu Tagliasacchi, Andrea Lindell, David B. Tulyakov, Sergey Computer Vision and Pattern Recognition Modern text-to-video synthesis models demonstrate coherent, photorealistic generation of complex videos from a text description. However, most existing models lack fine-grained control over camera movement, which is critical for downstream applications related to content creation, visual effects, and 3D vision. Recently, new methods demonstrate the ability to generate videos with controllable camera poses these techniques leverage pre-trained U-Net-based diffusion models that explicitly disentangle spatial and temporal generation. Still, no existing approach enables camera control for new, transformer-based video diffusion models that process spatial and temporal information jointly. Here, we propose to tame video transformers for 3D camera control using a ControlNet-like conditioning mechanism that incorporates spatiotemporal camera embeddings based on Plücker coordinates. The approach demonstrates state-of-the-art performance for controllable video generation after fine-tuning on the RealEstate10K dataset. To the best of our knowledge, our work is the first to enable camera control for transformer-based video diffusion models. |
| title | VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.12781 |