Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flow
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914284606324736 |
|---|---|
| author | Yu, Xinlei Xu, Chengming Zhang, Guibin He, Yongbo Chen, Zhangquan Xue, Zhucun Zhang, Jiangning Liao, Yue Hu, Xiaobin Jiang, Yu-Gang Yan, Shuicheng |
| author_facet | Yu, Xinlei Xu, Chengming Zhang, Guibin He, Yongbo Chen, Zhangquan Xue, Zhucun Zhang, Jiangning Liao, Yue Hu, Xiaobin Jiang, Yu-Gang Yan, Shuicheng |
| contents | Multi-Agent System (MAS) powered by Visual Language Models (VLMs) enables challenging tasks but suffers from a novel failure term, multi-agent visual hallucination snowballing, where hallucinations are seeded in a single agent and amplified by following ones due to the over-reliance on textual flow to relay visual information. Through turn-, layer-, and token-wise attention analyses, we provide detailed insights into the essence of hallucination snowballing regarding the reduction of visual attention allocation. It leads us to identify a subset of vision tokens with a unimodal attention peak in middle layers that best preserve visual evidence but gradually diminish in deeper agent turns, resulting in the visual hallucination snowballing in MAS. Thus, we propose ViF, a lightweight, plug-and-play mitigation paradigm that relays inter-agent messages with Visual Flow powered by the selected visual relay tokens and applies attention reallocation to amplify this pattern. The experiment results demonstrate that our method markedly reduces hallucination snowballing, consistently improving the performance across eight benchmarks based on four common MAS structures and ten base models. The source code is publicly available at: https://github.com/YU-deep/ViF.git. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_21789 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flow Yu, Xinlei Xu, Chengming Zhang, Guibin He, Yongbo Chen, Zhangquan Xue, Zhucun Zhang, Jiangning Liao, Yue Hu, Xiaobin Jiang, Yu-Gang Yan, Shuicheng Multiagent Systems Computer Vision and Pattern Recognition Multi-Agent System (MAS) powered by Visual Language Models (VLMs) enables challenging tasks but suffers from a novel failure term, multi-agent visual hallucination snowballing, where hallucinations are seeded in a single agent and amplified by following ones due to the over-reliance on textual flow to relay visual information. Through turn-, layer-, and token-wise attention analyses, we provide detailed insights into the essence of hallucination snowballing regarding the reduction of visual attention allocation. It leads us to identify a subset of vision tokens with a unimodal attention peak in middle layers that best preserve visual evidence but gradually diminish in deeper agent turns, resulting in the visual hallucination snowballing in MAS. Thus, we propose ViF, a lightweight, plug-and-play mitigation paradigm that relays inter-agent messages with Visual Flow powered by the selected visual relay tokens and applies attention reallocation to amplify this pattern. The experiment results demonstrate that our method markedly reduces hallucination snowballing, consistently improving the performance across eight benchmarks based on four common MAS structures and ten base models. The source code is publicly available at: https://github.com/YU-deep/ViF.git. |
| title | Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flow |
| topic | Multiagent Systems Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2509.21789 |