MTRDrive: Memory-Tool Synergistic Reasoning for Robust Autonomous Driving in Corner Cases
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909805820510208 |
|---|---|
| author | Luo, Ziang Qian, Kangan Wang, Jiahua Luo, Yuechen Miao, Jinyu Fu, Zheng Wang, Yunlong Jiang, Sicong Huang, Zilin Hu, Yifei Yang, Yuhao Ye, Hao Yang, Mengmeng Dong, Xiaojian Jiang, Kun Yang, Diange |
| author_facet | Luo, Ziang Qian, Kangan Wang, Jiahua Luo, Yuechen Miao, Jinyu Fu, Zheng Wang, Yunlong Jiang, Sicong Huang, Zilin Hu, Yifei Yang, Yuhao Ye, Hao Yang, Mengmeng Dong, Xiaojian Jiang, Kun Yang, Diange |
| contents | Vision-Language Models(VLMs) have demonstrated significant potential for end-to-end autonomous driving, yet a substantial gap remains between their current capabilities and the reliability necessary for real-world deployment. A critical challenge is their fragility, characterized by hallucinations and poor generalization in out-of-distribution (OOD) scenarios. To bridge this gap, we introduce MTRDrive, a novel framework that integrates procedural driving experiences with a dynamic toolkit to enhance generalization and proactive decision-making.
MTRDrive addresses these limitations through a closed-loop system that combines a memory-based experience retrieval mechanism with dynamic toolkits. This synergy enables the model to interact more effectively with its environment, improving both reasoning and decision-making capabilities with the help of our memory-tool synergistic reasoning. Additionally, we introduce a new benchmark based on complex Roadwork construction scenarios to rigorously evaluate zero-shot generalization.
Extensive experiments demonstrate the superior effectiveness of our approach. On the public NAVSIM benchmark, our 3B-parameter MTRDrive model achieves an exceptional PDMS of 88.3 without chain-of-thought and sets a state-of-the-art performance bar on high-level planning, with a driving metric score of 79.8\% and a planning accuracy of 82.6\%. Rigorous zero-shot evaluation on the new Roadwork-VLM benchmark shows a strong ability to reason robustly in unseen scenarios, achieving a driving metric score of 80.2\%. These results highlight MTRDrive's potential to advance autonomous driving toward safer and more reliable systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_20843 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MTRDrive: Memory-Tool Synergistic Reasoning for Robust Autonomous Driving in Corner Cases Luo, Ziang Qian, Kangan Wang, Jiahua Luo, Yuechen Miao, Jinyu Fu, Zheng Wang, Yunlong Jiang, Sicong Huang, Zilin Hu, Yifei Yang, Yuhao Ye, Hao Yang, Mengmeng Dong, Xiaojian Jiang, Kun Yang, Diange Robotics Vision-Language Models(VLMs) have demonstrated significant potential for end-to-end autonomous driving, yet a substantial gap remains between their current capabilities and the reliability necessary for real-world deployment. A critical challenge is their fragility, characterized by hallucinations and poor generalization in out-of-distribution (OOD) scenarios. To bridge this gap, we introduce MTRDrive, a novel framework that integrates procedural driving experiences with a dynamic toolkit to enhance generalization and proactive decision-making. MTRDrive addresses these limitations through a closed-loop system that combines a memory-based experience retrieval mechanism with dynamic toolkits. This synergy enables the model to interact more effectively with its environment, improving both reasoning and decision-making capabilities with the help of our memory-tool synergistic reasoning. Additionally, we introduce a new benchmark based on complex Roadwork construction scenarios to rigorously evaluate zero-shot generalization. Extensive experiments demonstrate the superior effectiveness of our approach. On the public NAVSIM benchmark, our 3B-parameter MTRDrive model achieves an exceptional PDMS of 88.3 without chain-of-thought and sets a state-of-the-art performance bar on high-level planning, with a driving metric score of 79.8\% and a planning accuracy of 82.6\%. Rigorous zero-shot evaluation on the new Roadwork-VLM benchmark shows a strong ability to reason robustly in unseen scenarios, achieving a driving metric score of 80.2\%. These results highlight MTRDrive's potential to advance autonomous driving toward safer and more reliable systems. |
| title | MTRDrive: Memory-Tool Synergistic Reasoning for Robust Autonomous Driving in Corner Cases |
| topic | Robotics |
| url | https://arxiv.org/abs/2509.20843 |