Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.05640 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915722270081024 |
|---|---|
| author | Li, Jingyu Wu, Junjie Hu, Dongnan Huang, Xiangkai Sun, Bin Hao, Zhihui Lang, Xianpeng Zhu, Xiatian Zhang, Li |
| author_facet | Li, Jingyu Wu, Junjie Hu, Dongnan Huang, Xiangkai Sun, Bin Hao, Zhihui Lang, Xianpeng Zhu, Xiatian Zhang, Li |
| contents | Recent end-to-end autonomous driving approaches have leveraged Vision-Language Models (VLMs) to enhance planning capabilities in complex driving scenarios. However, VLMs are inherently trained as generalist models, lacking specialized understanding of driving-specific reasoning in 3D space and time. When applied to autonomous driving, these models struggle to establish structured spatial-temporal representations that capture geometric relationships, scene context, and motion patterns critical for safe trajectory planning. To address these limitations, we propose SGDrive, a novel framework that explicitly structures the VLM's representation learning around driving-specific knowledge hierarchies. Built upon a pre-trained VLM backbone, SGDrive decomposes driving understanding into a scene-agent-goal hierarchy that mirrors human driving cognition: drivers first perceive the overall environment (scene context), then attend to safety-critical agents and their behaviors, and finally formulate short-term goals before executing actions. This hierarchical decomposition provides the structured spatial-temporal representation that generalist VLMs lack, integrating multi-level information into a compact yet comprehensive format for trajectory planning. Extensive experiments on the NAVSIM benchmark demonstrate that SGDrive achieves state-of-the-art performance among camera-only methods on both PDMS and EPDMS, validating the effectiveness of hierarchical knowledge structuring for adapting generalist VLMs to autonomous driving. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_05640 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Driving Li, Jingyu Wu, Junjie Hu, Dongnan Huang, Xiangkai Sun, Bin Hao, Zhihui Lang, Xianpeng Zhu, Xiatian Zhang, Li Computer Vision and Pattern Recognition Recent end-to-end autonomous driving approaches have leveraged Vision-Language Models (VLMs) to enhance planning capabilities in complex driving scenarios. However, VLMs are inherently trained as generalist models, lacking specialized understanding of driving-specific reasoning in 3D space and time. When applied to autonomous driving, these models struggle to establish structured spatial-temporal representations that capture geometric relationships, scene context, and motion patterns critical for safe trajectory planning. To address these limitations, we propose SGDrive, a novel framework that explicitly structures the VLM's representation learning around driving-specific knowledge hierarchies. Built upon a pre-trained VLM backbone, SGDrive decomposes driving understanding into a scene-agent-goal hierarchy that mirrors human driving cognition: drivers first perceive the overall environment (scene context), then attend to safety-critical agents and their behaviors, and finally formulate short-term goals before executing actions. This hierarchical decomposition provides the structured spatial-temporal representation that generalist VLMs lack, integrating multi-level information into a compact yet comprehensive format for trajectory planning. Extensive experiments on the NAVSIM benchmark demonstrate that SGDrive achieves state-of-the-art performance among camera-only methods on both PDMS and EPDMS, validating the effectiveness of hierarchical knowledge structuring for adapting generalist VLMs to autonomous driving. |
| title | SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Driving |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2601.05640 |