Unbiased Visual Reasoning with Controlled Visual Inputs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912790858432512 |
|---|---|
| author | Li, Zhaonan Lu, Shijie Wang, Fei Dineen, Jacob Ye, Xiao Xu, Zhikun Liu, Siyi Cho, Young Min Li, Bangzheng Chang, Daniel Nguyen, Kenny Yang, Qizheng Chen, Muhao Zhou, Ben |
| author_facet | Li, Zhaonan Lu, Shijie Wang, Fei Dineen, Jacob Ye, Xiao Xu, Zhikun Liu, Siyi Cho, Young Min Li, Bangzheng Chang, Daniel Nguyen, Kenny Yang, Qizheng Chen, Muhao Zhou, Ben |
| contents | End-to-end Vision-language Models (VLMs) often answer visual questions by exploiting spurious correlations instead of causal visual evidence, and can become more shortcut-prone when fine-tuned. We introduce VISTA (Visual-Information Separation for Text-based Analysis), a modular framework that decouples perception from reasoning via an explicit information bottleneck. A frozen VLM sensor is restricted to short, objective perception queries, while a text-only LLM reasoner decomposes each question, plans queries, and aggregates visual facts in natural language. This controlled interface defines a reward-aligned environment for training unbiased visual reasoning with reinforcement learning. Instantiated with Qwen2.5-VL and Llama3.2-Vision sensors, and trained with GRPO from only 641 curated multi-step questions, VISTA significantly improves robustness to real-world spurious correlations on SpuriVerse (+16.29% with Qwen-2.5-VL-7B and +6.77% with Llama-3.2-Vision-11B), while remaining competitive on MMVP and a balanced SeedBench subset. VISTA transfers robustly across unseen VLM sensors and is able to recognize and recover from VLM perception failures. Human analysis further shows that VISTA's reasoning traces are more neutral, less reliant on spurious attributes, and more explicitly grounded in visual evidence than end-to-end VLM baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_22183 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Unbiased Visual Reasoning with Controlled Visual Inputs Li, Zhaonan Lu, Shijie Wang, Fei Dineen, Jacob Ye, Xiao Xu, Zhikun Liu, Siyi Cho, Young Min Li, Bangzheng Chang, Daniel Nguyen, Kenny Yang, Qizheng Chen, Muhao Zhou, Ben Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language End-to-end Vision-language Models (VLMs) often answer visual questions by exploiting spurious correlations instead of causal visual evidence, and can become more shortcut-prone when fine-tuned. We introduce VISTA (Visual-Information Separation for Text-based Analysis), a modular framework that decouples perception from reasoning via an explicit information bottleneck. A frozen VLM sensor is restricted to short, objective perception queries, while a text-only LLM reasoner decomposes each question, plans queries, and aggregates visual facts in natural language. This controlled interface defines a reward-aligned environment for training unbiased visual reasoning with reinforcement learning. Instantiated with Qwen2.5-VL and Llama3.2-Vision sensors, and trained with GRPO from only 641 curated multi-step questions, VISTA significantly improves robustness to real-world spurious correlations on SpuriVerse (+16.29% with Qwen-2.5-VL-7B and +6.77% with Llama-3.2-Vision-11B), while remaining competitive on MMVP and a balanced SeedBench subset. VISTA transfers robustly across unseen VLM sensors and is able to recognize and recover from VLM perception failures. Human analysis further shows that VISTA's reasoning traces are more neutral, less reliant on spurious attributes, and more explicitly grounded in visual evidence than end-to-end VLM baselines. |
| title | Unbiased Visual Reasoning with Controlled Visual Inputs |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2512.22183 |