GesVLA: Gesture-Aware Vision-Language-Action Model Embedded Representations
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914588906225664 |
|---|---|
| author | Guo, Wenxuan Li, Ziyuan Zhang, Meng Liu, Yichen Dong, Yimeng Xu, Chuxi Wei, Yunfei Chen, Ze Zhou, Erjin Feng, Jianjiang |
| author_facet | Guo, Wenxuan Li, Ziyuan Zhang, Meng Liu, Yichen Dong, Yimeng Xu, Chuxi Wei, Yunfei Chen, Ze Zhou, Erjin Feng, Jianjiang |
| contents | Vision-Language-Action (VLA) models have shown strong potential for general-purpose robot manipulation by unifying perception and action. However, existing VLA systems primarily rely on textual instructions and struggle to resolve spatial ambiguity in complex scenes with multiple similar objects. To address this limitation, we introduce gesture as a parallel instruction modality and propose a Gesture-aware Vision-Language-Action model (GesVLA). Our approach encodes gesture features directly into the latent space, enabling them to participate in both high-level reasoning and low-level action generation, and adopts a dual-VLM architecture to achieve tight coupling between gesture representations and action policies. At the data level, we construct a scalable gesture data generation pipeline by rendering hand models onto real-world scene images. This reduces the sim-to-real visual gap while producing rich data with diverse motion patterns and corresponding pointing annotations. In addition, we employ a two-stage training strategy to equip the model with both gesture perception and action prediction capabilities. We evaluate our approach on multiple real-world robotic tasks, including a controlled block manipulation task for validation and more practical scenarios such as product and produce selection. Experimental results show that incorporating gesture consistently improves target grounding accuracy and human-robot interaction efficiency, especially in complex and cluttered environments. Project page: https://gwxuan.github.io/GesVLA/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_22812 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | GesVLA: Gesture-Aware Vision-Language-Action Model Embedded Representations Guo, Wenxuan Li, Ziyuan Zhang, Meng Liu, Yichen Dong, Yimeng Xu, Chuxi Wei, Yunfei Chen, Ze Zhou, Erjin Feng, Jianjiang Robotics Computer Vision and Pattern Recognition Vision-Language-Action (VLA) models have shown strong potential for general-purpose robot manipulation by unifying perception and action. However, existing VLA systems primarily rely on textual instructions and struggle to resolve spatial ambiguity in complex scenes with multiple similar objects. To address this limitation, we introduce gesture as a parallel instruction modality and propose a Gesture-aware Vision-Language-Action model (GesVLA). Our approach encodes gesture features directly into the latent space, enabling them to participate in both high-level reasoning and low-level action generation, and adopts a dual-VLM architecture to achieve tight coupling between gesture representations and action policies. At the data level, we construct a scalable gesture data generation pipeline by rendering hand models onto real-world scene images. This reduces the sim-to-real visual gap while producing rich data with diverse motion patterns and corresponding pointing annotations. In addition, we employ a two-stage training strategy to equip the model with both gesture perception and action prediction capabilities. We evaluate our approach on multiple real-world robotic tasks, including a controlled block manipulation task for validation and more practical scenarios such as product and produce selection. Experimental results show that incorporating gesture consistently improves target grounding accuracy and human-robot interaction efficiency, especially in complex and cluttered environments. Project page: https://gwxuan.github.io/GesVLA/. |
| title | GesVLA: Gesture-Aware Vision-Language-Action Model Embedded Representations |
| topic | Robotics Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2605.22812 |