Textual Steering Vectors Can Improve Visual Understanding in Multimodal Large Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908371469205504 |
|---|---|
| author | Gan, Woody Haosheng Fu, Deqing Asilis, Julian Liu, Ollie Yogatama, Dani Sharan, Vatsal Jia, Robin Neiswanger, Willie |
| author_facet | Gan, Woody Haosheng Fu, Deqing Asilis, Julian Liu, Ollie Yogatama, Dani Sharan, Vatsal Jia, Robin Neiswanger, Willie |
| contents | Steering methods have emerged as effective and targeted tools for guiding large language models' (LLMs) behavior without modifying their parameters. Multimodal large language models (MLLMs), however, do not currently enjoy the same suite of techniques, due in part to their recency and architectural diversity. Inspired by this gap, we investigate whether MLLMs can be steered using vectors derived from their text-only LLM backbone, via sparse autoencoders (SAEs), mean shift, and linear probing. We find that text-derived steering consistently enhances multimodal accuracy across diverse MLLM architectures and visual tasks. In particular, mean shift boosts spatial relationship accuracy on CV-Bench by up to +7.3% and counting accuracy by up to +3.3%, outperforming prompting and exhibiting strong generalization to out-of-distribution datasets. These results highlight textual steering vectors as a powerful, efficient mechanism for enhancing grounding in MLLMs with minimal additional data collection and computational overhead. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_14071 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Textual Steering Vectors Can Improve Visual Understanding in Multimodal Large Language Models Gan, Woody Haosheng Fu, Deqing Asilis, Julian Liu, Ollie Yogatama, Dani Sharan, Vatsal Jia, Robin Neiswanger, Willie Machine Learning Computation and Language Computer Vision and Pattern Recognition Steering methods have emerged as effective and targeted tools for guiding large language models' (LLMs) behavior without modifying their parameters. Multimodal large language models (MLLMs), however, do not currently enjoy the same suite of techniques, due in part to their recency and architectural diversity. Inspired by this gap, we investigate whether MLLMs can be steered using vectors derived from their text-only LLM backbone, via sparse autoencoders (SAEs), mean shift, and linear probing. We find that text-derived steering consistently enhances multimodal accuracy across diverse MLLM architectures and visual tasks. In particular, mean shift boosts spatial relationship accuracy on CV-Bench by up to +7.3% and counting accuracy by up to +3.3%, outperforming prompting and exhibiting strong generalization to out-of-distribution datasets. These results highlight textual steering vectors as a powerful, efficient mechanism for enhancing grounding in MLLMs with minimal additional data collection and computational overhead. |
| title | Textual Steering Vectors Can Improve Visual Understanding in Multimodal Large Language Models |
| topic | Machine Learning Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2505.14071 |