Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915761324294144 |
|---|---|
| author | Wang, Linhan Yang, Zichong Bai, Chen Zhang, Guoxiang Liu, Xiaotong Zheng, Xiaoyin Long, Xiao-Xiao Lu, Chang-Tien Lu, Cheng |
| author_facet | Wang, Linhan Yang, Zichong Bai, Chen Zhang, Guoxiang Liu, Xiaotong Zheng, Xiaoyin Long, Xiao-Xiao Lu, Chang-Tien Lu, Cheng |
| contents | End-to-end autonomous driving increasingly leverages self-supervised video pretraining to learn transferable planning representations. However, pretraining video world models for scene understanding has so far brought only limited improvements. This limitation is compounded by the inherent ambiguity of driving: each scene typically provides only a single human trajectory, making it difficult to learn multimodal behaviors. In this work, we propose Drive-JEPA, a framework that integrates Video Joint-Embedding Predictive Architecture (V-JEPA) with multimodal trajectory distillation for end-to-end driving. First, we adapt V-JEPA for end-to-end driving, pretraining a ViT encoder on large-scale driving videos to produce predictive representations aligned with trajectory planning. Second, we introduce a proposal-centric planner that distills diverse simulator-generated trajectories alongside human trajectories, with a momentum-aware selection mechanism to promote stable and safe behavior. When evaluated on NAVSIM, the V-JEPA representation combined with a simple transformer-based decoder outperforms prior methods by 3 PDMS in the perception-free setting. The complete Drive-JEPA framework achieves 93.3 PDMS on v1 and 87.8 EPDMS on v2, setting a new state-of-the-art. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_22032 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving Wang, Linhan Yang, Zichong Bai, Chen Zhang, Guoxiang Liu, Xiaotong Zheng, Xiaoyin Long, Xiao-Xiao Lu, Chang-Tien Lu, Cheng Computer Vision and Pattern Recognition End-to-end autonomous driving increasingly leverages self-supervised video pretraining to learn transferable planning representations. However, pretraining video world models for scene understanding has so far brought only limited improvements. This limitation is compounded by the inherent ambiguity of driving: each scene typically provides only a single human trajectory, making it difficult to learn multimodal behaviors. In this work, we propose Drive-JEPA, a framework that integrates Video Joint-Embedding Predictive Architecture (V-JEPA) with multimodal trajectory distillation for end-to-end driving. First, we adapt V-JEPA for end-to-end driving, pretraining a ViT encoder on large-scale driving videos to produce predictive representations aligned with trajectory planning. Second, we introduce a proposal-centric planner that distills diverse simulator-generated trajectories alongside human trajectories, with a momentum-aware selection mechanism to promote stable and safe behavior. When evaluated on NAVSIM, the V-JEPA representation combined with a simple transformer-based decoder outperforms prior methods by 3 PDMS in the perception-free setting. The complete Drive-JEPA framework achieves 93.3 PDMS on v1 and 87.8 EPDMS on v2, setting a new state-of-the-art. |
| title | Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2601.22032 |