Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrations
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915642888683520 |
|---|---|
| author | Mitra, Chancharik Luo, Yusen Saravanan, Raj Niu, Dantong Pai, Anirudh Thomason, Jesse Darrell, Trevor Anwar, Abrar Ramanan, Deva Herzig, Roei |
| author_facet | Mitra, Chancharik Luo, Yusen Saravanan, Raj Niu, Dantong Pai, Anirudh Thomason, Jesse Darrell, Trevor Anwar, Abrar Ramanan, Deva Herzig, Roei |
| contents | Vision-Language Action (VLAs) models promise to extend the remarkable success of vision-language models (VLMs) to robotics. Yet, unlike VLMs in the vision-language domain, VLAs for robotics require finetuning to contend with varying physical factors like robot embodiment, environment characteristics, and spatial relationships of each task. Existing fine-tuning methods lack specificity, adapting the same set of parameters regardless of a task's visual, linguistic, and physical characteristics. Inspired by functional specificity in neuroscience, we hypothesize that it is more effective to finetune sparse model representations specific to a given task. In this work, we introduce Robotic Steering, a finetuning approach grounded in mechanistic interpretability that leverages few-shot demonstrations to identify and selectively finetune task-specific attention heads aligned with the physical, visual, and linguistic requirements of robotic tasks. Through comprehensive on-robot evaluations with a Franka Emika robot arm, we demonstrate that Robotic Steering outperforms LoRA while achieving superior robustness under task variation, reduced computational cost, and enhanced interpretability for adapting VLAs to diverse robotic tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_22697 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrations Mitra, Chancharik Luo, Yusen Saravanan, Raj Niu, Dantong Pai, Anirudh Thomason, Jesse Darrell, Trevor Anwar, Abrar Ramanan, Deva Herzig, Roei Robotics Computation and Language Computer Vision and Pattern Recognition Vision-Language Action (VLAs) models promise to extend the remarkable success of vision-language models (VLMs) to robotics. Yet, unlike VLMs in the vision-language domain, VLAs for robotics require finetuning to contend with varying physical factors like robot embodiment, environment characteristics, and spatial relationships of each task. Existing fine-tuning methods lack specificity, adapting the same set of parameters regardless of a task's visual, linguistic, and physical characteristics. Inspired by functional specificity in neuroscience, we hypothesize that it is more effective to finetune sparse model representations specific to a given task. In this work, we introduce Robotic Steering, a finetuning approach grounded in mechanistic interpretability that leverages few-shot demonstrations to identify and selectively finetune task-specific attention heads aligned with the physical, visual, and linguistic requirements of robotic tasks. Through comprehensive on-robot evaluations with a Franka Emika robot arm, we demonstrate that Robotic Steering outperforms LoRA while achieving superior robustness under task variation, reduced computational cost, and enhanced interpretability for adapting VLAs to diverse robotic tasks. |
| title | Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrations |
| topic | Robotics Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2511.22697 |