Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2511.08246 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917073305731072 |
|---|---|
| author | Ma, Ziyu Gou, Chenhui Hu, Yiming Wang, Yong Chu, Xiangxiang Zhuang, Bohan Cai, Jianfei |
| author_facet | Ma, Ziyu Gou, Chenhui Hu, Yiming Wang, Yong Chu, Xiangxiang Zhuang, Bohan Cai, Jianfei |
| contents | Large Multimodal Models (LMMs) have shown promising in-context learning (ICL) capabilities, but scaling to many-shot settings remains difficult due to limited context length and high inference cost. To address these challenges, task-vector-based methods have been explored by inserting compact representations of many-shot in-context demonstrations into model activations. However, existing task-vector-based methods either overlook the importance of where to insert task vectors or struggle to determine suitable values for each location. To this end, we propose a novel Sensitivity-aware Task Vector insertion framework (STV) to figure out where and what to insert. Our key insight is that activation deltas across query-context pairs exhibit consistent structural patterns, providing a reliable cue for insertion. Based on the identified sensitive-aware locations, we construct a pre-clustered activation bank for each location by clustering the activation values, and then apply reinforcement learning to choose the most suitable one to insert. We evaluate STV across a range of multimodal models (e.g., Qwen-VL, Idefics-2) and tasks (e.g., VizWiz, OK-VQA), demonstrating its effectiveness and showing consistent improvements over previous task-vector-based methods with strong generalization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_08246 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Where and What Matters: Sensitivity-Aware Task Vectors for Many-Shot Multimodal In-Context Learning Ma, Ziyu Gou, Chenhui Hu, Yiming Wang, Yong Chu, Xiangxiang Zhuang, Bohan Cai, Jianfei Artificial Intelligence Large Multimodal Models (LMMs) have shown promising in-context learning (ICL) capabilities, but scaling to many-shot settings remains difficult due to limited context length and high inference cost. To address these challenges, task-vector-based methods have been explored by inserting compact representations of many-shot in-context demonstrations into model activations. However, existing task-vector-based methods either overlook the importance of where to insert task vectors or struggle to determine suitable values for each location. To this end, we propose a novel Sensitivity-aware Task Vector insertion framework (STV) to figure out where and what to insert. Our key insight is that activation deltas across query-context pairs exhibit consistent structural patterns, providing a reliable cue for insertion. Based on the identified sensitive-aware locations, we construct a pre-clustered activation bank for each location by clustering the activation values, and then apply reinforcement learning to choose the most suitable one to insert. We evaluate STV across a range of multimodal models (e.g., Qwen-VL, Idefics-2) and tasks (e.g., VizWiz, OK-VQA), demonstrating its effectiveness and showing consistent improvements over previous task-vector-based methods with strong generalization. |
| title | Where and What Matters: Sensitivity-Aware Task Vectors for Many-Shot Multimodal In-Context Learning |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2511.08246 |