Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918194854232064 |
|---|---|
| author | Bi, Jing Guo, Junjia Tang, Yunlong Wen, Lianggong Bruce Liu, Zhang Xu, Chenliang |
| author_facet | Bi, Jing Guo, Junjia Tang, Yunlong Wen, Lianggong Bruce Liu, Zhang Xu, Chenliang |
| contents | Recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated remarkable progress in visual understanding. This impressive leap raises a compelling question: how can language models, initially trained solely on linguistic data, effectively interpret and process visual content? This paper aims to address this question with systematic investigation across 4 model families and 4 model scales, uncovering a unique class of attention heads that focus specifically on visual content. Our analysis reveals a strong correlation between the behavior of these attention heads, the distribution of attention weights, and their concentration on visual tokens within the input. These findings enhance our understanding of how LLMs adapt to multimodal tasks, demonstrating their potential to bridge the gap between textual and visual understanding. This work paves the way for the development of AI systems capable of engaging with diverse modalities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_18108 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach Bi, Jing Guo, Junjia Tang, Yunlong Wen, Lianggong Bruce Liu, Zhang Xu, Chenliang Computer Vision and Pattern Recognition Recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated remarkable progress in visual understanding. This impressive leap raises a compelling question: how can language models, initially trained solely on linguistic data, effectively interpret and process visual content? This paper aims to address this question with systematic investigation across 4 model families and 4 model scales, uncovering a unique class of attention heads that focus specifically on visual content. Our analysis reveals a strong correlation between the behavior of these attention heads, the distribution of attention weights, and their concentration on visual tokens within the input. These findings enhance our understanding of how LLMs adapt to multimodal tasks, demonstrating their potential to bridge the gap between textual and visual understanding. This work paves the way for the development of AI systems capable of engaging with diverse modalities. |
| title | Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.18108 |