LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trends
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914423292035072 |
|---|---|
| author | Cui, Can Ma, Yunsheng Park, Sung-Yeon Yang, Zichong Zhou, Yupeng Liu, Peiran Lu, Juanwu Peng, Juntong Zhang, Jiaru Zhang, Ruqi Li, Lingxi Chen, Yaobin Panchal, Jitesh H. Abdelraouf, Amr Gupta, Rohit Han, Kyungtae Wang, Ziran |
| author_facet | Cui, Can Ma, Yunsheng Park, Sung-Yeon Yang, Zichong Zhou, Yupeng Liu, Peiran Lu, Juanwu Peng, Juntong Zhang, Jiaru Zhang, Ruqi Li, Lingxi Chen, Yaobin Panchal, Jitesh H. Abdelraouf, Amr Gupta, Rohit Han, Kyungtae Wang, Ziran |
| contents | With the broader adoption and highly successful development of Large Language Models (LLMs), there has been growing interest and demand for applying LLMs to autonomous driving technology. Driven by their natural language understanding and reasoning capabilities, LLMs have the potential to enhance various aspects of autonomous driving systems, from perception and scene understanding to interactive decision-making. This paper first introduces the novel concept of designing Large Language Models for Autonomous Driving (LLM4AD), followed by a review of existing LLM4AD studies. Then, a comprehensive benchmark is proposed for evaluating the instruction-following and reasoning abilities of LLM4AD systems, which includes LaMPilot-Bench, CARLA Leaderboard 1.0 Benchmark in simulation and NuPlanQA for multi-view visual question answering. Furthermore, extensive real-world experiments are conducted on autonomous vehicle platforms, examining both on-cloud and on-edge LLM deployment for personalized decision-making and motion control. Next, the future trends of integrating language diffusion models into autonomous driving are explored, exemplified by the proposed ViLaD (Vision-Language Diffusion) framework. Finally, the main challenges of LLM4AD are discussed, including latency, deployment, security and privacy, safety, trust and transparency, and personalization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_15281 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trends Cui, Can Ma, Yunsheng Park, Sung-Yeon Yang, Zichong Zhou, Yupeng Liu, Peiran Lu, Juanwu Peng, Juntong Zhang, Jiaru Zhang, Ruqi Li, Lingxi Chen, Yaobin Panchal, Jitesh H. Abdelraouf, Amr Gupta, Rohit Han, Kyungtae Wang, Ziran Robotics Artificial Intelligence Computation and Language Human-Computer Interaction With the broader adoption and highly successful development of Large Language Models (LLMs), there has been growing interest and demand for applying LLMs to autonomous driving technology. Driven by their natural language understanding and reasoning capabilities, LLMs have the potential to enhance various aspects of autonomous driving systems, from perception and scene understanding to interactive decision-making. This paper first introduces the novel concept of designing Large Language Models for Autonomous Driving (LLM4AD), followed by a review of existing LLM4AD studies. Then, a comprehensive benchmark is proposed for evaluating the instruction-following and reasoning abilities of LLM4AD systems, which includes LaMPilot-Bench, CARLA Leaderboard 1.0 Benchmark in simulation and NuPlanQA for multi-view visual question answering. Furthermore, extensive real-world experiments are conducted on autonomous vehicle platforms, examining both on-cloud and on-edge LLM deployment for personalized decision-making and motion control. Next, the future trends of integrating language diffusion models into autonomous driving are explored, exemplified by the proposed ViLaD (Vision-Language Diffusion) framework. Finally, the main challenges of LLM4AD are discussed, including latency, deployment, security and privacy, safety, trust and transparency, and personalization. |
| title | LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trends |
| topic | Robotics Artificial Intelligence Computation and Language Human-Computer Interaction |
| url | https://arxiv.org/abs/2410.15281 |