Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.04678 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909017280872448 |
|---|---|
| author | Lin, Yihan Li, Haoyang Li, Yang Shen, Haitao Zhao, Yihan Shao, Chao Zhang, Jing |
| author_facet | Lin, Yihan Li, Haoyang Li, Yang Shen, Haitao Zhao, Yihan Shao, Chao Zhang, Jing |
| contents | Latent actions serve as an intermediate representation that enables consistent modeling of vision-language-action (VLA) models across heterogeneous datasets. However, approaches to supervising VLAs with latent actions are fragmented and lack a systematic comparison. This work structures the study of latent action supervision from two perspectives: (i) regularizing the trajectory via image-based latent actions, and (ii) unifying the target space with action-based latent actions. Under a unified VLA baseline, we instantiate and compare four representative integration strategies. Our results reveal a formulation-task correspondence: image-based latent actions benefit long-horizon reasoning and scene-level generalization, whereas action-based latent actions excel at complex motor coordination. Furthermore, we find that directly supervising the VLM with discrete latent action tokens yields the most effective performance. Finally, our experiments offer initial insights into the benefits of latent action supervision in mixed-data, suggesting a promising direction for VLA training. Code is available at https://github.com/RUCKBReasoning/From_Pixels_to_Tokens. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_04678 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | From Pixels to Tokens: A Systematic Study of Latent Action Supervision for Vision-Language-Action Models Lin, Yihan Li, Haoyang Li, Yang Shen, Haitao Zhao, Yihan Shao, Chao Zhang, Jing Robotics Computer Vision and Pattern Recognition Latent actions serve as an intermediate representation that enables consistent modeling of vision-language-action (VLA) models across heterogeneous datasets. However, approaches to supervising VLAs with latent actions are fragmented and lack a systematic comparison. This work structures the study of latent action supervision from two perspectives: (i) regularizing the trajectory via image-based latent actions, and (ii) unifying the target space with action-based latent actions. Under a unified VLA baseline, we instantiate and compare four representative integration strategies. Our results reveal a formulation-task correspondence: image-based latent actions benefit long-horizon reasoning and scene-level generalization, whereas action-based latent actions excel at complex motor coordination. Furthermore, we find that directly supervising the VLM with discrete latent action tokens yields the most effective performance. Finally, our experiments offer initial insights into the benefits of latent action supervision in mixed-data, suggesting a promising direction for VLA training. Code is available at https://github.com/RUCKBReasoning/From_Pixels_to_Tokens. |
| title | From Pixels to Tokens: A Systematic Study of Latent Action Supervision for Vision-Language-Action Models |
| topic | Robotics Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2605.04678 |