The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912862264360960 |
|---|---|
| author | Serra, Alessandro Pietro Ortu, Francesco Panizon, Emanuele Valeriani, Lucrezia Basile, Lorenzo Ansuini, Alessio Doimo, Diego Cazzaniga, Alberto |
| author_facet | Serra, Alessandro Pietro Ortu, Francesco Panizon, Emanuele Valeriani, Lucrezia Basile, Lorenzo Ansuini, Alessio Doimo, Diego Cazzaniga, Alberto |
| contents | Recent advances in multimodal training have significantly improved the integration of image understanding and generation within a unified model. This study investigates how vision-language models (VLMs) handle image-understanding tasks, focusing on how visual information is processed and transferred to the textual domain. We compare native multimodal VLMs, models trained from scratch on multimodal data to generate both text and images, and non-native multimodal VLMs, models adapted from pre-trained large language models or capable of generating only text, highlighting key differences in information flow. We find that in native multimodal VLMs, image and text embeddings are more separated within the residual stream. Moreover, VLMs differ in how visual information reaches text: non-native multimodal VLMs exhibit a distributed communication pattern, where information is exchanged through multiple image tokens, whereas models trained natively for joint image and text generation tend to rely on a single post-image token that acts as a narrow gate for visual information. We show that ablating this single token significantly deteriorates image-understanding performance, whereas targeted, token-level interventions reliably steer image semantics and downstream text with fine-grained control. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_06646 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models Serra, Alessandro Pietro Ortu, Francesco Panizon, Emanuele Valeriani, Lucrezia Basile, Lorenzo Ansuini, Alessio Doimo, Diego Cazzaniga, Alberto Computer Vision and Pattern Recognition Machine Learning Recent advances in multimodal training have significantly improved the integration of image understanding and generation within a unified model. This study investigates how vision-language models (VLMs) handle image-understanding tasks, focusing on how visual information is processed and transferred to the textual domain. We compare native multimodal VLMs, models trained from scratch on multimodal data to generate both text and images, and non-native multimodal VLMs, models adapted from pre-trained large language models or capable of generating only text, highlighting key differences in information flow. We find that in native multimodal VLMs, image and text embeddings are more separated within the residual stream. Moreover, VLMs differ in how visual information reaches text: non-native multimodal VLMs exhibit a distributed communication pattern, where information is exchanged through multiple image tokens, whereas models trained natively for joint image and text generation tend to rely on a single post-image token that acts as a narrow gate for visual information. We show that ablating this single token significantly deteriorates image-understanding performance, whereas targeted, token-level interventions reliably steer image semantics and downstream text with fine-grained control. |
| title | The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2412.06646 |