DiVE: DiT-based Video Generation with Enhanced Control
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910588322447360 |
|---|---|
| author | Jiang, Junpeng Hong, Gangyi Zhou, Lijun Ma, Enhui Hu, Hengtong Zhou, Xia Xiang, Jie Liu, Fan Yu, Kaicheng Sun, Haiyang Zhan, Kun Jia, Peng Zhang, Miao |
| author_facet | Jiang, Junpeng Hong, Gangyi Zhou, Lijun Ma, Enhui Hu, Hengtong Zhou, Xia Xiang, Jie Liu, Fan Yu, Kaicheng Sun, Haiyang Zhan, Kun Jia, Peng Zhang, Miao |
| contents | Generating high-fidelity, temporally consistent videos in autonomous driving scenarios faces a significant challenge, e.g. problematic maneuvers in corner cases. Despite recent video generation works are proposed to tackcle the mentioned problem, i.e. models built on top of Diffusion Transformers (DiT), works are still missing which are targeted on exploring the potential for multi-view videos generation scenarios. Noticeably, we propose the first DiT-based framework specifically designed for generating temporally and multi-view consistent videos which precisely match the given bird's-eye view layouts control. Specifically, the proposed framework leverages a parameter-free spatial view-inflated attention mechanism to guarantee the cross-view consistency, where joint cross-attention modules and ControlNet-Transformer are integrated to further improve the precision of control. To demonstrate our advantages, we extensively investigate the qualitative comparisons on nuScenes dataset, particularly in some most challenging corner cases. In summary, the effectiveness of our proposed method in producing long, controllable, and highly consistent videos under difficult conditions is proven to be effective. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_01595 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DiVE: DiT-based Video Generation with Enhanced Control Jiang, Junpeng Hong, Gangyi Zhou, Lijun Ma, Enhui Hu, Hengtong Zhou, Xia Xiang, Jie Liu, Fan Yu, Kaicheng Sun, Haiyang Zhan, Kun Jia, Peng Zhang, Miao Computer Vision and Pattern Recognition Generating high-fidelity, temporally consistent videos in autonomous driving scenarios faces a significant challenge, e.g. problematic maneuvers in corner cases. Despite recent video generation works are proposed to tackcle the mentioned problem, i.e. models built on top of Diffusion Transformers (DiT), works are still missing which are targeted on exploring the potential for multi-view videos generation scenarios. Noticeably, we propose the first DiT-based framework specifically designed for generating temporally and multi-view consistent videos which precisely match the given bird's-eye view layouts control. Specifically, the proposed framework leverages a parameter-free spatial view-inflated attention mechanism to guarantee the cross-view consistency, where joint cross-attention modules and ControlNet-Transformer are integrated to further improve the precision of control. To demonstrate our advantages, we extensively investigate the qualitative comparisons on nuScenes dataset, particularly in some most challenging corner cases. In summary, the effectiveness of our proposed method in producing long, controllable, and highly consistent videos under difficult conditions is proven to be effective. |
| title | DiVE: DiT-based Video Generation with Enhanced Control |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2409.01595 |