Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.03739 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917312471236608 |
|---|---|
| author | Fan, Zehua Lyu, Wenqi Song, Wenxuan Zhao, Linge Yang, Yifei Wang, Xi He, Junjie Huang, Lida Liu, Haiyan Sun, Bingchuan Bao, Guangjun Mao, Xuanyao Xu, Liang Wang, Yan Gao, Feng |
| author_facet | Fan, Zehua Lyu, Wenqi Song, Wenxuan Zhao, Linge Yang, Yifei Wang, Xi He, Junjie Huang, Lida Liu, Haiyan Sun, Bingchuan Bao, Guangjun Mao, Xuanyao Xu, Liang Wang, Yan Gao, Feng |
| contents | Multimodal large language models (MLLMs) have advanced zero-shot end-to-end Vision-Language Navigation (VLN), yet robust navigation requires not only semantic understanding but also predictive modeling of environment dynamics and spatial structure. We propose PROSPECT, a unified streaming navigation agent that couples a streaming Vision-Language-Action (VLA) policy with latent predictive representation learning. PROSPECT uses CUT3R as a streaming 3D foundation spatial encoder to produce long-context, absolute-scale spatial features, and fuses them with SigLIP semantic features via cross-attention. During training, we introduce learnable stream query tokens that query the streaming context and predict next-step 2D and 3D latent features (rather than pixels or explicit modalities), supervised in the latent spaces of frozen SigLIP and CUT3R teachers. The predictive branch shapes internal representations without inference overhead. Experiments on VLN-CE benchmarks and real-robot deployment demonstrate state-of-the-art performance and improved long-horizon robustness under diverse lighting. We will release code for the community soon. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_03739 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | PROSPECT: Unified Streaming Vision-Language Navigation via Semantic--Spatial Fusion and Latent Predictive Representation Fan, Zehua Lyu, Wenqi Song, Wenxuan Zhao, Linge Yang, Yifei Wang, Xi He, Junjie Huang, Lida Liu, Haiyan Sun, Bingchuan Bao, Guangjun Mao, Xuanyao Xu, Liang Wang, Yan Gao, Feng Computer Vision and Pattern Recognition Artificial Intelligence Multimodal large language models (MLLMs) have advanced zero-shot end-to-end Vision-Language Navigation (VLN), yet robust navigation requires not only semantic understanding but also predictive modeling of environment dynamics and spatial structure. We propose PROSPECT, a unified streaming navigation agent that couples a streaming Vision-Language-Action (VLA) policy with latent predictive representation learning. PROSPECT uses CUT3R as a streaming 3D foundation spatial encoder to produce long-context, absolute-scale spatial features, and fuses them with SigLIP semantic features via cross-attention. During training, we introduce learnable stream query tokens that query the streaming context and predict next-step 2D and 3D latent features (rather than pixels or explicit modalities), supervised in the latent spaces of frozen SigLIP and CUT3R teachers. The predictive branch shapes internal representations without inference overhead. Experiments on VLN-CE benchmarks and real-robot deployment demonstrate state-of-the-art performance and improved long-horizon robustness under diverse lighting. We will release code for the community soon. |
| title | PROSPECT: Unified Streaming Vision-Language Navigation via Semantic--Spatial Fusion and Latent Predictive Representation |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2603.03739 |