EMMA: End-to-End Multimodal Model for Autonomous Driving
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912600408719360 |
|---|---|
| author | Hwang, Jyh-Jing Xu, Runsheng Lin, Hubert Hung, Wei-Chih Ji, Jingwei Choi, Kristy Huang, Di He, Tong Covington, Paul Sapp, Benjamin Zhou, Yin Guo, James Anguelov, Dragomir Tan, Mingxing |
| author_facet | Hwang, Jyh-Jing Xu, Runsheng Lin, Hubert Hung, Wei-Chih Ji, Jingwei Choi, Kristy Huang, Di He, Tong Covington, Paul Sapp, Benjamin Zhou, Yin Guo, James Anguelov, Dragomir Tan, Mingxing |
| contents | We introduce EMMA, an End-to-end Multimodal Model for Autonomous driving. Built upon a multi-modal large language model foundation like Gemini, EMMA directly maps raw camera sensor data into various driving-specific outputs, including planner trajectories, perception objects, and road graph elements. EMMA maximizes the utility of world knowledge from the pre-trained large language models, by representing all non-sensor inputs (e.g. navigation instructions and ego vehicle status) and outputs (e.g. trajectories and 3D locations) as natural language text. This approach allows EMMA to jointly process various driving tasks in a unified language space, and generate the outputs for each task using task-specific prompts. Empirically, we demonstrate EMMA's effectiveness by achieving state-of-the-art performance in motion planning on nuScenes as well as competitive results on the Waymo Open Motion Dataset (WOMD). EMMA also yields competitive results for camera-primary 3D object detection on the Waymo Open Dataset (WOD). We show that co-training EMMA with planner trajectories, object detection, and road graph tasks yields improvements across all three domains, highlighting EMMA's potential as a generalist model for autonomous driving applications. We hope that our results will inspire research to further evolve the state of the art in autonomous driving model architectures. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_23262 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | EMMA: End-to-End Multimodal Model for Autonomous Driving Hwang, Jyh-Jing Xu, Runsheng Lin, Hubert Hung, Wei-Chih Ji, Jingwei Choi, Kristy Huang, Di He, Tong Covington, Paul Sapp, Benjamin Zhou, Yin Guo, James Anguelov, Dragomir Tan, Mingxing Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning Robotics We introduce EMMA, an End-to-end Multimodal Model for Autonomous driving. Built upon a multi-modal large language model foundation like Gemini, EMMA directly maps raw camera sensor data into various driving-specific outputs, including planner trajectories, perception objects, and road graph elements. EMMA maximizes the utility of world knowledge from the pre-trained large language models, by representing all non-sensor inputs (e.g. navigation instructions and ego vehicle status) and outputs (e.g. trajectories and 3D locations) as natural language text. This approach allows EMMA to jointly process various driving tasks in a unified language space, and generate the outputs for each task using task-specific prompts. Empirically, we demonstrate EMMA's effectiveness by achieving state-of-the-art performance in motion planning on nuScenes as well as competitive results on the Waymo Open Motion Dataset (WOMD). EMMA also yields competitive results for camera-primary 3D object detection on the Waymo Open Dataset (WOD). We show that co-training EMMA with planner trajectories, object detection, and road graph tasks yields improvements across all three domains, highlighting EMMA's potential as a generalist model for autonomous driving applications. We hope that our results will inspire research to further evolve the state of the art in autonomous driving model architectures. |
| title | EMMA: End-to-End Multimodal Model for Autonomous Driving |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning Robotics |
| url | https://arxiv.org/abs/2410.23262 |