DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913404225060864 |
|---|---|
| author | Tian, Xiaoyu Gu, Junru Li, Bailin Liu, Yicheng Wang, Yang Zhao, Zhiyong Zhan, Kun Jia, Peng Lang, Xianpeng Zhao, Hang |
| author_facet | Tian, Xiaoyu Gu, Junru Li, Bailin Liu, Yicheng Wang, Yang Zhao, Zhiyong Zhan, Kun Jia, Peng Lang, Xianpeng Zhao, Hang |
| contents | A primary hurdle of autonomous driving in urban environments is understanding complex and long-tail scenarios, such as challenging road conditions and delicate human behaviors. We introduce DriveVLM, an autonomous driving system leveraging Vision-Language Models (VLMs) for enhanced scene understanding and planning capabilities. DriveVLM integrates a unique combination of reasoning modules for scene description, scene analysis, and hierarchical planning. Furthermore, recognizing the limitations of VLMs in spatial reasoning and heavy computational requirements, we propose DriveVLM-Dual, a hybrid system that synergizes the strengths of DriveVLM with the traditional autonomous driving pipeline. Experiments on both the nuScenes dataset and our SUP-AD dataset demonstrate the efficacy of DriveVLM and DriveVLM-Dual in handling complex and unpredictable driving conditions. Finally, we deploy the DriveVLM-Dual on a production vehicle, verifying it is effective in real-world autonomous driving environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_12289 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models Tian, Xiaoyu Gu, Junru Li, Bailin Liu, Yicheng Wang, Yang Zhao, Zhiyong Zhan, Kun Jia, Peng Lang, Xianpeng Zhao, Hang Computer Vision and Pattern Recognition A primary hurdle of autonomous driving in urban environments is understanding complex and long-tail scenarios, such as challenging road conditions and delicate human behaviors. We introduce DriveVLM, an autonomous driving system leveraging Vision-Language Models (VLMs) for enhanced scene understanding and planning capabilities. DriveVLM integrates a unique combination of reasoning modules for scene description, scene analysis, and hierarchical planning. Furthermore, recognizing the limitations of VLMs in spatial reasoning and heavy computational requirements, we propose DriveVLM-Dual, a hybrid system that synergizes the strengths of DriveVLM with the traditional autonomous driving pipeline. Experiments on both the nuScenes dataset and our SUP-AD dataset demonstrate the efficacy of DriveVLM and DriveVLM-Dual in handling complex and unpredictable driving conditions. Finally, we deploy the DriveVLM-Dual on a production vehicle, verifying it is effective in real-world autonomous driving environments. |
| title | DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2402.12289 |