CLAP: Contrastive Latent Action Pretraining for Learning Vision-Language-Action Models from Human Videos
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912808072904704 |
|---|---|
| author | Zhang, Chubin Wang, Jianan Gao, Zifeng Su, Yue Dai, Tianru Zhou, Cai Lu, Jiwen Tang, Yansong |
| author_facet | Zhang, Chubin Wang, Jianan Gao, Zifeng Su, Yue Dai, Tianru Zhou, Cai Lu, Jiwen Tang, Yansong |
| contents | Generalist Vision-Language-Action models are currently hindered by the scarcity of robotic data compared to the abundance of human video demonstrations. Existing Latent Action Models attempt to leverage video data but often suffer from visual entanglement, capturing noise rather than manipulation skills. To address this, we propose Contrastive Latent Action Pretraining (CLAP), a framework that aligns the visual latent space from videos with a proprioceptive latent space from robot trajectories. By employing contrastive learning, CLAP maps video transitions onto a quantized, physically executable codebook. Building on this representation, we introduce a dual-formulation VLA framework offering both CLAP-NTP, an autoregressive model excelling at instruction following and object generalization, and CLAP-RF, a Rectified Flow-based policy designed for high-frequency, precise manipulation. Furthermore, we propose a Knowledge Matching (KM) regularization strategy to mitigate catastrophic forgetting during fine-tuning. Extensive experiments demonstrate that CLAP significantly outperforms strong baselines, enabling the effective transfer of skills from human videos to robotic execution. Project page: https://lin-shan.com/CLAP/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_04061 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | CLAP: Contrastive Latent Action Pretraining for Learning Vision-Language-Action Models from Human Videos Zhang, Chubin Wang, Jianan Gao, Zifeng Su, Yue Dai, Tianru Zhou, Cai Lu, Jiwen Tang, Yansong Robotics Computer Vision and Pattern Recognition Generalist Vision-Language-Action models are currently hindered by the scarcity of robotic data compared to the abundance of human video demonstrations. Existing Latent Action Models attempt to leverage video data but often suffer from visual entanglement, capturing noise rather than manipulation skills. To address this, we propose Contrastive Latent Action Pretraining (CLAP), a framework that aligns the visual latent space from videos with a proprioceptive latent space from robot trajectories. By employing contrastive learning, CLAP maps video transitions onto a quantized, physically executable codebook. Building on this representation, we introduce a dual-formulation VLA framework offering both CLAP-NTP, an autoregressive model excelling at instruction following and object generalization, and CLAP-RF, a Rectified Flow-based policy designed for high-frequency, precise manipulation. Furthermore, we propose a Knowledge Matching (KM) regularization strategy to mitigate catastrophic forgetting during fine-tuning. Extensive experiments demonstrate that CLAP significantly outperforms strong baselines, enabling the effective transfer of skills from human videos to robotic execution. Project page: https://lin-shan.com/CLAP/. |
| title | CLAP: Contrastive Latent Action Pretraining for Learning Vision-Language-Action Models from Human Videos |
| topic | Robotics Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2601.04061 |