BEVWorld: A Multimodal World Simulator for Autonomous Driving via Scene-Level BEV Latents
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915266100723712 |
|---|---|
| author | Zhang, Yumeng Gong, Shi Xiong, Kaixin Ye, Xiaoqing Li, Xiaofan Tan, Xiao Wang, Fan Huang, Jizhou Wu, Hua Wang, Haifeng |
| author_facet | Zhang, Yumeng Gong, Shi Xiong, Kaixin Ye, Xiaoqing Li, Xiaofan Tan, Xiao Wang, Fan Huang, Jizhou Wu, Hua Wang, Haifeng |
| contents | World models have attracted increasing attention in autonomous driving for their ability to forecast potential future scenarios. In this paper, we propose BEVWorld, a novel framework that transforms multimodal sensor inputs into a unified and compact Bird's Eye View (BEV) latent space for holistic environment modeling. The proposed world model consists of two main components: a multi-modal tokenizer and a latent BEV sequence diffusion model. The multi-modal tokenizer first encodes heterogeneous sensory data, and its decoder reconstructs the latent BEV tokens into LiDAR and surround-view image observations via ray-casting rendering in a self-supervised manner. This enables joint modeling and bidirectional encoding-decoding of panoramic imagery and point cloud data within a shared spatial representation. On top of this, the latent BEV sequence diffusion model performs temporally consistent forecasting of future scenes, conditioned on high-level action tokens, enabling scene-level reasoning over time. Extensive experiments demonstrate the effectiveness of BEVWorld on autonomous driving benchmarks, showcasing its capability in realistic future scene generation and its benefits for downstream tasks such as perception and motion prediction. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_05679 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | BEVWorld: A Multimodal World Simulator for Autonomous Driving via Scene-Level BEV Latents Zhang, Yumeng Gong, Shi Xiong, Kaixin Ye, Xiaoqing Li, Xiaofan Tan, Xiao Wang, Fan Huang, Jizhou Wu, Hua Wang, Haifeng Computer Vision and Pattern Recognition Artificial Intelligence World models have attracted increasing attention in autonomous driving for their ability to forecast potential future scenarios. In this paper, we propose BEVWorld, a novel framework that transforms multimodal sensor inputs into a unified and compact Bird's Eye View (BEV) latent space for holistic environment modeling. The proposed world model consists of two main components: a multi-modal tokenizer and a latent BEV sequence diffusion model. The multi-modal tokenizer first encodes heterogeneous sensory data, and its decoder reconstructs the latent BEV tokens into LiDAR and surround-view image observations via ray-casting rendering in a self-supervised manner. This enables joint modeling and bidirectional encoding-decoding of panoramic imagery and point cloud data within a shared spatial representation. On top of this, the latent BEV sequence diffusion model performs temporally consistent forecasting of future scenes, conditioned on high-level action tokens, enabling scene-level reasoning over time. Extensive experiments demonstrate the effectiveness of BEVWorld on autonomous driving benchmarks, showcasing its capability in realistic future scene generation and its benefits for downstream tasks such as perception and motion prediction. |
| title | BEVWorld: A Multimodal World Simulator for Autonomous Driving via Scene-Level BEV Latents |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2407.05679 |