DriveScape: Towards High-Resolution Controllable Multi-View Driving Video Generation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910600989245440 |
|---|---|
| author | Wu, Wei Guo, Xi Tang, Weixuan Huang, Tingxuan Wang, Chiyu Chen, Dongyue Ding, Chenjing |
| author_facet | Wu, Wei Guo, Xi Tang, Weixuan Huang, Tingxuan Wang, Chiyu Chen, Dongyue Ding, Chenjing |
| contents | Recent advancements in generative models have provided promising solutions for synthesizing realistic driving videos, which are crucial for training autonomous driving perception models. However, existing approaches often struggle with multi-view video generation due to the challenges of integrating 3D information while maintaining spatial-temporal consistency and effectively learning from a unified model. We propose DriveScape, an end-to-end framework for multi-view, 3D condition-guided video generation, capable of producing 1024 x 576 high-resolution videos at 10Hz. Unlike other methods limited to 2Hz due to the 3D box annotation frame rate, DriveScape overcomes this with its ability to operate under sparse conditions. Our Bi-Directional Modulated Transformer (BiMot) ensures precise alignment of 3D structural information, maintaining spatial-temporal consistency. DriveScape excels in video generation performance, achieving state-of-the-art results on the nuScenes dataset with an FID score of 8.34 and an FVD score of 76.39. Our project homepage: https://metadrivescape.github.io/papers_project/drivescapev1/index.html |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_05463 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DriveScape: Towards High-Resolution Controllable Multi-View Driving Video Generation Wu, Wei Guo, Xi Tang, Weixuan Huang, Tingxuan Wang, Chiyu Chen, Dongyue Ding, Chenjing Computer Vision and Pattern Recognition Recent advancements in generative models have provided promising solutions for synthesizing realistic driving videos, which are crucial for training autonomous driving perception models. However, existing approaches often struggle with multi-view video generation due to the challenges of integrating 3D information while maintaining spatial-temporal consistency and effectively learning from a unified model. We propose DriveScape, an end-to-end framework for multi-view, 3D condition-guided video generation, capable of producing 1024 x 576 high-resolution videos at 10Hz. Unlike other methods limited to 2Hz due to the 3D box annotation frame rate, DriveScape overcomes this with its ability to operate under sparse conditions. Our Bi-Directional Modulated Transformer (BiMot) ensures precise alignment of 3D structural information, maintaining spatial-temporal consistency. DriveScape excels in video generation performance, achieving state-of-the-art results on the nuScenes dataset with an FID score of 8.34 and an FVD score of 76.39. Our project homepage: https://metadrivescape.github.io/papers_project/drivescapev1/index.html |
| title | DriveScape: Towards High-Resolution Controllable Multi-View Driving Video Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2409.05463 |