Distilling Multi-modal Large Language Models for Autonomous Driving
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929678664597504 |
|---|---|
| author | Hegde, Deepti Yasarla, Rajeev Cai, Hong Han, Shizhong Bhattacharyya, Apratim Mahajan, Shweta Liu, Litian Garrepalli, Risheek Patel, Vishal M. Porikli, Fatih |
| author_facet | Hegde, Deepti Yasarla, Rajeev Cai, Hong Han, Shizhong Bhattacharyya, Apratim Mahajan, Shweta Liu, Litian Garrepalli, Risheek Patel, Vishal M. Porikli, Fatih |
| contents | Autonomous driving demands safe motion planning, especially in critical "long-tail" scenarios. Recent end-to-end autonomous driving systems leverage large language models (LLMs) as planners to improve generalizability to rare events. However, using LLMs at test time introduces high computational costs. To address this, we propose DiMA, an end-to-end autonomous driving system that maintains the efficiency of an LLM-free (or vision-based) planner while leveraging the world knowledge of an LLM. DiMA distills the information from a multi-modal LLM to a vision-based end-to-end planner through a set of specially designed surrogate tasks. Under a joint training strategy, a scene encoder common to both networks produces structured representations that are semantically grounded as well as aligned to the final planning objective. Notably, the LLM is optional at inference, enabling robust planning without compromising on efficiency. Training with DiMA results in a 37% reduction in the L2 trajectory error and an 80% reduction in the collision rate of the vision-based planner, as well as a 44% trajectory error reduction in longtail scenarios. DiMA also achieves state-of-the-art performance on the nuScenes planning benchmark. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_09757 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Distilling Multi-modal Large Language Models for Autonomous Driving Hegde, Deepti Yasarla, Rajeev Cai, Hong Han, Shizhong Bhattacharyya, Apratim Mahajan, Shweta Liu, Litian Garrepalli, Risheek Patel, Vishal M. Porikli, Fatih Computer Vision and Pattern Recognition Robotics Autonomous driving demands safe motion planning, especially in critical "long-tail" scenarios. Recent end-to-end autonomous driving systems leverage large language models (LLMs) as planners to improve generalizability to rare events. However, using LLMs at test time introduces high computational costs. To address this, we propose DiMA, an end-to-end autonomous driving system that maintains the efficiency of an LLM-free (or vision-based) planner while leveraging the world knowledge of an LLM. DiMA distills the information from a multi-modal LLM to a vision-based end-to-end planner through a set of specially designed surrogate tasks. Under a joint training strategy, a scene encoder common to both networks produces structured representations that are semantically grounded as well as aligned to the final planning objective. Notably, the LLM is optional at inference, enabling robust planning without compromising on efficiency. Training with DiMA results in a 37% reduction in the L2 trajectory error and an 80% reduction in the collision rate of the vision-based planner, as well as a 44% trajectory error reduction in longtail scenarios. DiMA also achieves state-of-the-art performance on the nuScenes planning benchmark. |
| title | Distilling Multi-modal Large Language Models for Autonomous Driving |
| topic | Computer Vision and Pattern Recognition Robotics |
| url | https://arxiv.org/abs/2501.09757 |