Do VLMs Perceive or Recall? Probing Visual Perception vs. Memory with Classic Visual Illusions
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908922935246848 |
|---|---|
| author | Sun, Xiaoxiao Li, Mingyang Yuan, Kun Sun, Min Woo Endo, Mark Wu, Shengguang Li, Changlin Zhang, Yuhui Wang, Zeyu Yeung-Levy, Serena |
| author_facet | Sun, Xiaoxiao Li, Mingyang Yuan, Kun Sun, Min Woo Endo, Mark Wu, Shengguang Li, Changlin Zhang, Yuhui Wang, Zeyu Yeung-Levy, Serena |
| contents | Large Vision-Language Models (VLMs) often answer classic visual illusions "correctly" on original images, yet persist with the same responses when illusion factors are inverted, even though the visual change is obvious to humans. This raises a fundamental question: do VLMs perceive visual changes or merely recall memorized patterns? While several studies have noted this phenomenon, the underlying causes remain unclear. To move from observations to systematic understanding, this paper introduces VI-Probe, a controllable visual-illusion framework with graded perturbations and matched visual controls (without illusion inducer) that disentangles visually grounded perception from language-driven recall. Unlike prior work that focuses on averaged accuracy, we measure stability and sensitivity using Polarity-Flip Consistency, Template Fixation Index, and an illusion multiplier normalized against matched controls. Experiments across different families reveal that response persistence arises from heterogeneous causes rather than a single mechanism. For instance, GPT-5 exhibits memory override, Claude-Opus-4.1 shows perception-memory competition, while Qwen variants suggest visual-processing limits. Our findings challenge single-cause views and motivate probing-based evaluation that measures both knowledge and sensitivity to controlled visual change. Data and code are available at https://sites.google.com/view/vi-probe/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_22150 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Do VLMs Perceive or Recall? Probing Visual Perception vs. Memory with Classic Visual Illusions Sun, Xiaoxiao Li, Mingyang Yuan, Kun Sun, Min Woo Endo, Mark Wu, Shengguang Li, Changlin Zhang, Yuhui Wang, Zeyu Yeung-Levy, Serena Computer Vision and Pattern Recognition Large Vision-Language Models (VLMs) often answer classic visual illusions "correctly" on original images, yet persist with the same responses when illusion factors are inverted, even though the visual change is obvious to humans. This raises a fundamental question: do VLMs perceive visual changes or merely recall memorized patterns? While several studies have noted this phenomenon, the underlying causes remain unclear. To move from observations to systematic understanding, this paper introduces VI-Probe, a controllable visual-illusion framework with graded perturbations and matched visual controls (without illusion inducer) that disentangles visually grounded perception from language-driven recall. Unlike prior work that focuses on averaged accuracy, we measure stability and sensitivity using Polarity-Flip Consistency, Template Fixation Index, and an illusion multiplier normalized against matched controls. Experiments across different families reveal that response persistence arises from heterogeneous causes rather than a single mechanism. For instance, GPT-5 exhibits memory override, Claude-Opus-4.1 shows perception-memory competition, while Qwen variants suggest visual-processing limits. Our findings challenge single-cause views and motivate probing-based evaluation that measures both knowledge and sensitivity to controlled visual change. Data and code are available at https://sites.google.com/view/vi-probe/ |
| title | Do VLMs Perceive or Recall? Probing Visual Perception vs. Memory with Classic Visual Illusions |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2601.22150 |