RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Driving
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912524069240832 |
|---|---|
| author | Huang, Zhijian Feng, Chengjian Yan, Feng Xiao, Baihui Jie, Zequn Zhong, Yujie Liang, Xiaodan Ma, Lin |
| author_facet | Huang, Zhijian Feng, Chengjian Yan, Feng Xiao, Baihui Jie, Zequn Zhong, Yujie Liang, Xiaodan Ma, Lin |
| contents | Large Multimodal Models (LMMs) have demonstrated exceptional comprehension and interpretation capabilities in Autonomous Driving (AD) by incorporating large language models. Despite the advancements, current data-driven AD approaches tend to concentrate on a single dataset and specific tasks, neglecting their overall capabilities and ability to generalize. To bridge these gaps, we propose RoboTron-Drive, a general large multimodal model designed to process diverse data inputs, such as images and multi-view videos, while performing a broad spectrum of AD tasks, including perception, prediction, and planning. Initially, the model undergoes curriculum pre-training to process varied visual signals and perform basic visual comprehension and perception tasks. Subsequently, we augment and standardize various AD datasets to finetune the model, resulting in an all-in-one LMM for autonomous driving. To assess the general capabilities and generalization ability, we conduct evaluations on six public benchmarks and undertake zero-shot transfer on three unseen datasets, where RoboTron-Drive achieves state-of-the-art performance across all tasks. We hope RoboTron-Drive as a promising solution for AD in the real world. Project page with code: https://github.com/zhijian11/RoboTron-Drive. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_07689 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Driving Huang, Zhijian Feng, Chengjian Yan, Feng Xiao, Baihui Jie, Zequn Zhong, Yujie Liang, Xiaodan Ma, Lin Computer Vision and Pattern Recognition Multimedia Robotics Large Multimodal Models (LMMs) have demonstrated exceptional comprehension and interpretation capabilities in Autonomous Driving (AD) by incorporating large language models. Despite the advancements, current data-driven AD approaches tend to concentrate on a single dataset and specific tasks, neglecting their overall capabilities and ability to generalize. To bridge these gaps, we propose RoboTron-Drive, a general large multimodal model designed to process diverse data inputs, such as images and multi-view videos, while performing a broad spectrum of AD tasks, including perception, prediction, and planning. Initially, the model undergoes curriculum pre-training to process varied visual signals and perform basic visual comprehension and perception tasks. Subsequently, we augment and standardize various AD datasets to finetune the model, resulting in an all-in-one LMM for autonomous driving. To assess the general capabilities and generalization ability, we conduct evaluations on six public benchmarks and undertake zero-shot transfer on three unseen datasets, where RoboTron-Drive achieves state-of-the-art performance across all tasks. We hope RoboTron-Drive as a promising solution for AD in the real world. Project page with code: https://github.com/zhijian11/RoboTron-Drive. |
| title | RoboTron-Drive: All-in-One Large Multimodal Model for Autonomous Driving |
| topic | Computer Vision and Pattern Recognition Multimedia Robotics |
| url | https://arxiv.org/abs/2412.07689 |