Spatial-Aware VLA Pretraining through Visual-Physical Alignment from Human Videos
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912764764618752 |
|---|---|
| author | Feng, Yicheng Zhang, Wanpeng Wang, Ye Luo, Hao Yuan, Haoqi Zheng, Sipeng Lu, Zongqing |
| author_facet | Feng, Yicheng Zhang, Wanpeng Wang, Ye Luo, Hao Yuan, Haoqi Zheng, Sipeng Lu, Zongqing |
| contents | Vision-Language-Action (VLA) models provide a promising paradigm for robot learning by integrating visual perception with language-guided policy learning. However, most existing approaches rely on 2D visual inputs to perform actions in 3D physical environments, creating a significant gap between perception and action grounding. To bridge this gap, we propose a Spatial-Aware VLA Pretraining paradigm that performs explicit alignment between visual space and physical space during pretraining, enabling models to acquire 3D spatial understanding before robot policy learning. Starting from pretrained vision-language models, we leverage large-scale human demonstration videos to extract 3D visual and 3D action annotations, forming a new source of supervision that aligns 2D visual observations with 3D spatial reasoning. We instantiate this paradigm with VIPA-VLA, a dual-encoder architecture that incorporates a 3D visual encoder to augment semantic visual representations with 3D-aware features. When adapted to downstream robot tasks, VIPA-VLA achieves significantly improved grounding between 2D vision and 3D action, resulting in more robust and generalizable robotic policies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_13080 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Spatial-Aware VLA Pretraining through Visual-Physical Alignment from Human Videos Feng, Yicheng Zhang, Wanpeng Wang, Ye Luo, Hao Yuan, Haoqi Zheng, Sipeng Lu, Zongqing Robotics Vision-Language-Action (VLA) models provide a promising paradigm for robot learning by integrating visual perception with language-guided policy learning. However, most existing approaches rely on 2D visual inputs to perform actions in 3D physical environments, creating a significant gap between perception and action grounding. To bridge this gap, we propose a Spatial-Aware VLA Pretraining paradigm that performs explicit alignment between visual space and physical space during pretraining, enabling models to acquire 3D spatial understanding before robot policy learning. Starting from pretrained vision-language models, we leverage large-scale human demonstration videos to extract 3D visual and 3D action annotations, forming a new source of supervision that aligns 2D visual observations with 3D spatial reasoning. We instantiate this paradigm with VIPA-VLA, a dual-encoder architecture that incorporates a 3D visual encoder to augment semantic visual representations with 3D-aware features. When adapted to downstream robot tasks, VIPA-VLA achieves significantly improved grounding between 2D vision and 3D action, resulting in more robust and generalizable robotic policies. |
| title | Spatial-Aware VLA Pretraining through Visual-Physical Alignment from Human Videos |
| topic | Robotics |
| url | https://arxiv.org/abs/2512.13080 |