VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910011928608768 |
|---|---|
| author | Yu, Xinlei Xu, Chengming Zhang, Guibin Chen, Zhangquan Zhang, Yudong He, Yongbo Jiang, Peng-Tao Zhang, Jiangning Hu, Xiaobin Yan, Shuicheng |
| author_facet | Yu, Xinlei Xu, Chengming Zhang, Guibin Chen, Zhangquan Zhang, Yudong He, Yongbo Jiang, Peng-Tao Zhang, Jiangning Hu, Xiaobin Yan, Shuicheng |
| contents | Despite the remarkable success of Vision-Language Models (VLMs), their performance on a range of complex visual tasks is often hindered by a "visual processing bottleneck": a propensity to lose grounding in visual evidence and exhibit a deficit in contextualized visual experience during prolonged generation. Drawing inspiration from human cognitive memory theory, which distinguishes short-term visually-dominant memory and long-term semantically-dominant memory, we propose VisMem, a cognitively-aligned framework that equips VLMs with dynamic latent vision memories, a short-term module for fine-grained perceptual retention and a long-term module for abstract semantic consolidation. These memories are seamlessly invoked during inference, allowing VLMs to maintain both perceptual fidelity and semantic consistency across thinking and generation. Extensive experiments across diverse visual benchmarks for understanding, reasoning, and generation reveal that VisMem delivers a significant average performance boost of 11.0% relative to the vanilla model and outperforms all counterparts, establishing a new paradigm for latent-space memory enhancement. The code will be available: https://github.com/YU-deep/VisMem.git. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_11007 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models Yu, Xinlei Xu, Chengming Zhang, Guibin Chen, Zhangquan Zhang, Yudong He, Yongbo Jiang, Peng-Tao Zhang, Jiangning Hu, Xiaobin Yan, Shuicheng Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Despite the remarkable success of Vision-Language Models (VLMs), their performance on a range of complex visual tasks is often hindered by a "visual processing bottleneck": a propensity to lose grounding in visual evidence and exhibit a deficit in contextualized visual experience during prolonged generation. Drawing inspiration from human cognitive memory theory, which distinguishes short-term visually-dominant memory and long-term semantically-dominant memory, we propose VisMem, a cognitively-aligned framework that equips VLMs with dynamic latent vision memories, a short-term module for fine-grained perceptual retention and a long-term module for abstract semantic consolidation. These memories are seamlessly invoked during inference, allowing VLMs to maintain both perceptual fidelity and semantic consistency across thinking and generation. Extensive experiments across diverse visual benchmarks for understanding, reasoning, and generation reveal that VisMem delivers a significant average performance boost of 11.0% relative to the vanilla model and outperforms all counterparts, establishing a new paradigm for latent-space memory enhancement. The code will be available: https://github.com/YU-deep/VisMem.git. |
| title | VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2511.11007 |