iGVLM: Dynamic Instruction-Guided Vision Encoding for Question-Aware Multimodal Understanding
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910045810196480 |
|---|---|
| author | Liu, Hanpeng Li, Yaqian Wang, Zidan Zhang, Shuoxi Bo, Zihao Takezoe, Rinyoichi Long, Kaiwen He, Kun |
| author_facet | Liu, Hanpeng Li, Yaqian Wang, Zidan Zhang, Shuoxi Bo, Zihao Takezoe, Rinyoichi Long, Kaiwen He, Kun |
| contents | Despite the success of Large Vision--Language Models (LVLMs), most existing architectures suffer from a representation bottleneck: they rely on static, instruction-agnostic vision encoders whose visual representations are utilized in an invariant manner across different textual tasks. This rigidity hinders fine-grained reasoning where task-specific visual cues are critical. To address this issue, we propose iGVLM, a general framework for instruction-guided visual modulation. iGVLM introduces a decoupled dual-branch architecture: a frozen representation branch that preserves task-agnostic visual representations learned during pre-training, and a dynamic conditioning branch that performs affine feature modulation via Adaptive Layer Normalization (AdaLN). This design enables a smooth transition from general-purpose perception to instruction-aware reasoning while maintaining the structural integrity and stability of pre-trained visual priors. Beyond standard benchmarks, we introduce MM4, a controlled diagnostic probe for quantifying logical consistency under multi-query, multi-instruction settings. Extensive results show that iGVLM consistently enhances instruction sensitivity across diverse language backbones, offering a plug-and-play paradigm for bridging passive perception and active reasoning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_02748 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | iGVLM: Dynamic Instruction-Guided Vision Encoding for Question-Aware Multimodal Understanding Liu, Hanpeng Li, Yaqian Wang, Zidan Zhang, Shuoxi Bo, Zihao Takezoe, Rinyoichi Long, Kaiwen He, Kun Computer Vision and Pattern Recognition Artificial Intelligence Despite the success of Large Vision--Language Models (LVLMs), most existing architectures suffer from a representation bottleneck: they rely on static, instruction-agnostic vision encoders whose visual representations are utilized in an invariant manner across different textual tasks. This rigidity hinders fine-grained reasoning where task-specific visual cues are critical. To address this issue, we propose iGVLM, a general framework for instruction-guided visual modulation. iGVLM introduces a decoupled dual-branch architecture: a frozen representation branch that preserves task-agnostic visual representations learned during pre-training, and a dynamic conditioning branch that performs affine feature modulation via Adaptive Layer Normalization (AdaLN). This design enables a smooth transition from general-purpose perception to instruction-aware reasoning while maintaining the structural integrity and stability of pre-trained visual priors. Beyond standard benchmarks, we introduce MM4, a controlled diagnostic probe for quantifying logical consistency under multi-query, multi-instruction settings. Extensive results show that iGVLM consistently enhances instruction sensitivity across diverse language backbones, offering a plug-and-play paradigm for bridging passive perception and active reasoning. |
| title | iGVLM: Dynamic Instruction-Guided Vision Encoding for Question-Aware Multimodal Understanding |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2603.02748 |