Confusion-Aware In-Context-Learning for Vision-Language Models in Robotic Manipulation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917346923249664 |
|---|---|
| author | He, Yayun Kang, Zuheng Zhao, Botao Wu, Zhouyin Peng, Junqing Wang, Jianzong |
| author_facet | He, Yayun Kang, Zuheng Zhao, Botao Wu, Zhouyin Peng, Junqing Wang, Jianzong |
| contents | Vision-language models (VLMs) have significantly improved the generalization capabilities of robotic manipulation. However, VLM-based systems often suffer from a lack of robustness, leading to unpredictable errors, particularly in scenarios involving confusable objects. Our preliminary analysis reveals that these failures are mainly caused by shortcut learning problem inherently in VLMs, limiting their ability to accurately distinguish between confusable features. To this end, we propose Confusion-Aware In-Context Learning (CAICL), a method that enhances VLM performance in confusable scenarios for robotic manipulation. The approach begins with confusion localization and analysis, identifying potential sources of confusion. This information is then used as a prompt for the VLM to focus on features most likely to cause misidentification. Extensive experiments on the VIMA-Bench show that CAICL effectively addresses the shortcut learning issue, achieving a 85.5\% success rate and showing good stability across tasks with different degrees of generalization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_15134 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Confusion-Aware In-Context-Learning for Vision-Language Models in Robotic Manipulation He, Yayun Kang, Zuheng Zhao, Botao Wu, Zhouyin Peng, Junqing Wang, Jianzong Robotics Vision-language models (VLMs) have significantly improved the generalization capabilities of robotic manipulation. However, VLM-based systems often suffer from a lack of robustness, leading to unpredictable errors, particularly in scenarios involving confusable objects. Our preliminary analysis reveals that these failures are mainly caused by shortcut learning problem inherently in VLMs, limiting their ability to accurately distinguish between confusable features. To this end, we propose Confusion-Aware In-Context Learning (CAICL), a method that enhances VLM performance in confusable scenarios for robotic manipulation. The approach begins with confusion localization and analysis, identifying potential sources of confusion. This information is then used as a prompt for the VLM to focus on features most likely to cause misidentification. Extensive experiments on the VIMA-Bench show that CAICL effectively addresses the shortcut learning issue, achieving a 85.5\% success rate and showing good stability across tasks with different degrees of generalization. |
| title | Confusion-Aware In-Context-Learning for Vision-Language Models in Robotic Manipulation |
| topic | Robotics |
| url | https://arxiv.org/abs/2603.15134 |