Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.25922 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916044860293120 |
|---|---|
| author | Liu, Xiao Liu, Jiaxiang Peng, Boci Hu, Boren Wang, Yusong Chen, Xiwen Tiwari, Prayag Zhang, Liming Xu, Mingkun |
| author_facet | Liu, Xiao Liu, Jiaxiang Peng, Boci Hu, Boren Wang, Yusong Chen, Xiwen Tiwari, Prayag Zhang, Liming Xu, Mingkun |
| contents | Vision Language Models adapt well to downstream tasks but are highly vulnerable to adversarial perturbations that disrupt cross-modal semantic alignment. Existing defenses are largely unidirectional or structural, failing to exploit bidirectional cross-modal complementarity and instance-wise adaptive protection. To overcome the limitations of unidirectional and static defenses in adversarial settings, we propose Closed-Loop Bidirectional Prompting, casting robust adaptation as cross-modal agreement recovery via a dynamic feedback loop on frozen encoders. A Semantic Anchor is introduced as a stable prior to constrain cyclic updates and mitigate perturbation-induced feature corruption. Through anchor-based bootstrapping, textual semantics denoise visual representations, while the refined visuals enable instance-adaptive prompt updating, yielding a rectified and robust consensus. Extensive evaluations across 11 datasets validate state-of-the-art robustness and strong base-to-new generalization, while maintaining a favorable trade-off between computational cost and accuracy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_25922 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Closed-Loop Bidirectional Prompting for Adversarial Robustness of Vision Language Models Liu, Xiao Liu, Jiaxiang Peng, Boci Hu, Boren Wang, Yusong Chen, Xiwen Tiwari, Prayag Zhang, Liming Xu, Mingkun Computer Vision and Pattern Recognition Vision Language Models adapt well to downstream tasks but are highly vulnerable to adversarial perturbations that disrupt cross-modal semantic alignment. Existing defenses are largely unidirectional or structural, failing to exploit bidirectional cross-modal complementarity and instance-wise adaptive protection. To overcome the limitations of unidirectional and static defenses in adversarial settings, we propose Closed-Loop Bidirectional Prompting, casting robust adaptation as cross-modal agreement recovery via a dynamic feedback loop on frozen encoders. A Semantic Anchor is introduced as a stable prior to constrain cyclic updates and mitigate perturbation-induced feature corruption. Through anchor-based bootstrapping, textual semantics denoise visual representations, while the refined visuals enable instance-adaptive prompt updating, yielding a rectified and robust consensus. Extensive evaluations across 11 datasets validate state-of-the-art robustness and strong base-to-new generalization, while maintaining a favorable trade-off between computational cost and accuracy. |
| title | Closed-Loop Bidirectional Prompting for Adversarial Robustness of Vision Language Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2605.25922 |