Vision-Language Models Unlock Task-Centric Latent Actions
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914294693625856 |
|---|---|
| author | Nikulin, Alexander Zisman, Ilya Klepach, Albina Tarasov, Denis Derevyagin, Alexander Polubarov, Andrei Nikita, Lyubaykin Kurenkov, Vladislav |
| author_facet | Nikulin, Alexander Zisman, Ilya Klepach, Albina Tarasov, Denis Derevyagin, Alexander Polubarov, Andrei Nikita, Lyubaykin Kurenkov, Vladislav |
| contents | Latent Action Models (LAMs) have rapidly gained traction as an important component in the pre-training pipelines of leading Vision-Language-Action models. However, they fail when observations contain action-correlated distractors, often encoding noise instead of meaningful latent actions. Humans, on the other hand, can effortlessly distinguish task-relevant motions from irrelevant details in any video given only a brief task description. In this work, we propose to utilize the common-sense reasoning abilities of Vision-Language Models (VLMs) to provide promptable representations, effectively separating controllable changes from the noise in unsupervised way. We use these representations as targets during LAM training and benchmark a wide variety of popular VLMs, revealing substantial variation in the quality of promptable representations as well as their robustness to different prompts and hyperparameters. Interestingly, we find that more recent VLMs may perform worse than older ones. Finally, we show that simply asking VLMs to ignore distractors can substantially improve latent action quality, yielding up to a six-fold increase in downstream success rates on Distracting MetaWorld. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_22714 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Vision-Language Models Unlock Task-Centric Latent Actions Nikulin, Alexander Zisman, Ilya Klepach, Albina Tarasov, Denis Derevyagin, Alexander Polubarov, Andrei Nikita, Lyubaykin Kurenkov, Vladislav Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Latent Action Models (LAMs) have rapidly gained traction as an important component in the pre-training pipelines of leading Vision-Language-Action models. However, they fail when observations contain action-correlated distractors, often encoding noise instead of meaningful latent actions. Humans, on the other hand, can effortlessly distinguish task-relevant motions from irrelevant details in any video given only a brief task description. In this work, we propose to utilize the common-sense reasoning abilities of Vision-Language Models (VLMs) to provide promptable representations, effectively separating controllable changes from the noise in unsupervised way. We use these representations as targets during LAM training and benchmark a wide variety of popular VLMs, revealing substantial variation in the quality of promptable representations as well as their robustness to different prompts and hyperparameters. Interestingly, we find that more recent VLMs may perform worse than older ones. Finally, we show that simply asking VLMs to ignore distractors can substantially improve latent action quality, yielding up to a six-fold increase in downstream success rates on Distracting MetaWorld. |
| title | Vision-Language Models Unlock Task-Centric Latent Actions |
| topic | Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2601.22714 |