RenderWorld: World Model with Self-Supervised 3D Label
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909490253660160 |
|---|---|
| author | Yan, Ziyang Dong, Wenzhen Shao, Yihua Lu, Yuhang Haiyang, Liu Liu, Jingwen Wang, Haozhe Wang, Zhe Wang, Yan Remondino, Fabio Ma, Yuexin |
| author_facet | Yan, Ziyang Dong, Wenzhen Shao, Yihua Lu, Yuhang Haiyang, Liu Liu, Jingwen Wang, Haozhe Wang, Zhe Wang, Yan Remondino, Fabio Ma, Yuexin |
| contents | End-to-end autonomous driving with vision-only is not only more cost-effective compared to LiDAR-vision fusion but also more reliable than traditional methods. To achieve a economical and robust purely visual autonomous driving system, we propose RenderWorld, a vision-only end-to-end autonomous driving framework, which generates 3D occupancy labels using a self-supervised gaussian-based Img2Occ Module, then encodes the labels by AM-VAE, and uses world model for forecasting and planning. RenderWorld employs Gaussian Splatting to represent 3D scenes and render 2D images greatly improves segmentation accuracy and reduces GPU memory consumption compared with NeRF-based methods. By applying AM-VAE to encode air and non-air separately, RenderWorld achieves more fine-grained scene element representation, leading to state-of-the-art performance in both 4D occupancy forecasting and motion planning from autoregressive world model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_11356 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | RenderWorld: World Model with Self-Supervised 3D Label Yan, Ziyang Dong, Wenzhen Shao, Yihua Lu, Yuhang Haiyang, Liu Liu, Jingwen Wang, Haozhe Wang, Zhe Wang, Yan Remondino, Fabio Ma, Yuexin Computer Vision and Pattern Recognition Artificial Intelligence End-to-end autonomous driving with vision-only is not only more cost-effective compared to LiDAR-vision fusion but also more reliable than traditional methods. To achieve a economical and robust purely visual autonomous driving system, we propose RenderWorld, a vision-only end-to-end autonomous driving framework, which generates 3D occupancy labels using a self-supervised gaussian-based Img2Occ Module, then encodes the labels by AM-VAE, and uses world model for forecasting and planning. RenderWorld employs Gaussian Splatting to represent 3D scenes and render 2D images greatly improves segmentation accuracy and reduces GPU memory consumption compared with NeRF-based methods. By applying AM-VAE to encode air and non-air separately, RenderWorld achieves more fine-grained scene element representation, leading to state-of-the-art performance in both 4D occupancy forecasting and motion planning from autoregressive world model. |
| title | RenderWorld: World Model with Self-Supervised 3D Label |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2409.11356 |