Is a 3D-Tokenized LLM the Key to Reliable Autonomous Driving?
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913367163142144 |
|---|---|
| author | Bai, Yifan Wu, Dongming Liu, Yingfei Jia, Fan Mao, Weixin Zhang, Ziheng Zhao, Yucheng Shen, Jianbing Wei, Xing Wang, Tiancai Zhang, Xiangyu |
| author_facet | Bai, Yifan Wu, Dongming Liu, Yingfei Jia, Fan Mao, Weixin Zhang, Ziheng Zhao, Yucheng Shen, Jianbing Wei, Xing Wang, Tiancai Zhang, Xiangyu |
| contents | Rapid advancements in Autonomous Driving (AD) tasks turned a significant shift toward end-to-end fashion, particularly in the utilization of vision-language models (VLMs) that integrate robust logical reasoning and cognitive abilities to enable comprehensive end-to-end planning. However, these VLM-based approaches tend to integrate 2D vision tokenizers and a large language model (LLM) for ego-car planning, which lack 3D geometric priors as a cornerstone of reliable planning. Naturally, this observation raises a critical concern: Can a 2D-tokenized LLM accurately perceive the 3D environment? Our evaluation of current VLM-based methods across 3D object detection, vectorized map construction, and environmental caption suggests that the answer is, unfortunately, NO. In other words, 2D-tokenized LLM fails to provide reliable autonomous driving. In response, we introduce DETR-style 3D perceptrons as 3D tokenizers, which connect LLM with a one-layer linear projector. This simple yet elegant strategy, termed Atlas, harnesses the inherent priors of the 3D physical world, enabling it to simultaneously process high-resolution multi-view images and employ spatiotemporal modeling. Despite its simplicity, Atlas demonstrates superior performance in both 3D detection and ego planning tasks on nuScenes dataset, proving that 3D-tokenized LLM is the key to reliable autonomous driving. The code and datasets will be released. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_18361 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Is a 3D-Tokenized LLM the Key to Reliable Autonomous Driving? Bai, Yifan Wu, Dongming Liu, Yingfei Jia, Fan Mao, Weixin Zhang, Ziheng Zhao, Yucheng Shen, Jianbing Wei, Xing Wang, Tiancai Zhang, Xiangyu Computer Vision and Pattern Recognition Rapid advancements in Autonomous Driving (AD) tasks turned a significant shift toward end-to-end fashion, particularly in the utilization of vision-language models (VLMs) that integrate robust logical reasoning and cognitive abilities to enable comprehensive end-to-end planning. However, these VLM-based approaches tend to integrate 2D vision tokenizers and a large language model (LLM) for ego-car planning, which lack 3D geometric priors as a cornerstone of reliable planning. Naturally, this observation raises a critical concern: Can a 2D-tokenized LLM accurately perceive the 3D environment? Our evaluation of current VLM-based methods across 3D object detection, vectorized map construction, and environmental caption suggests that the answer is, unfortunately, NO. In other words, 2D-tokenized LLM fails to provide reliable autonomous driving. In response, we introduce DETR-style 3D perceptrons as 3D tokenizers, which connect LLM with a one-layer linear projector. This simple yet elegant strategy, termed Atlas, harnesses the inherent priors of the 3D physical world, enabling it to simultaneously process high-resolution multi-view images and employ spatiotemporal modeling. Despite its simplicity, Atlas demonstrates superior performance in both 3D detection and ego planning tasks on nuScenes dataset, proving that 3D-tokenized LLM is the key to reliable autonomous driving. The code and datasets will be released. |
| title | Is a 3D-Tokenized LLM the Key to Reliable Autonomous Driving? |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2405.18361 |