Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.06502 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910110245191680 |
|---|---|
| author | Qi, Peigui Tang, Kunsheng Yu, Yanpu Wu, Jialin Song, Yide Zhou, Wenbo Huang, Zhicong Hong, Cheng Zhang, Weiming Yu, Nenghai |
| author_facet | Qi, Peigui Tang, Kunsheng Yu, Yanpu Wu, Jialin Song, Yide Zhou, Wenbo Huang, Zhicong Hong, Cheng Zhang, Weiming Yu, Nenghai |
| contents | Vision-Language Models (VLMs) face significant safety vulnerabilities from malicious prompt attacks due to weakened alignment during visual integration. Existing defenses suffer from efficiency and robustness. To address these challenges, we first propose the Multimodal Aggregated Feature Extraction (MAFE) framework that enables CLIP to handle long text and fuse multimodal information into unified representations. Through empirical analysis of MAFE-extracted features, we discover distinct distributional patterns between benign and malicious prompts. Building upon this finding, we develop VLMShield, a lightweight safety detector that efficiently identifies multimodal malicious attacks as a plug-and-play solution. Extensive experiments demonstrate superior performance across multiple dimensions, including robustness, efficiency, and utility. Through our work, we hope to pave the way for more secure multimodal AI deployment. Code is available at [this https URL](https://github.com/pgqihere/VLMShield). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_06502 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | VLMShield: Efficient and Robust Defense of Vision-Language Models against Malicious Prompts Qi, Peigui Tang, Kunsheng Yu, Yanpu Wu, Jialin Song, Yide Zhou, Wenbo Huang, Zhicong Hong, Cheng Zhang, Weiming Yu, Nenghai Machine Learning I.2; K.7 Vision-Language Models (VLMs) face significant safety vulnerabilities from malicious prompt attacks due to weakened alignment during visual integration. Existing defenses suffer from efficiency and robustness. To address these challenges, we first propose the Multimodal Aggregated Feature Extraction (MAFE) framework that enables CLIP to handle long text and fuse multimodal information into unified representations. Through empirical analysis of MAFE-extracted features, we discover distinct distributional patterns between benign and malicious prompts. Building upon this finding, we develop VLMShield, a lightweight safety detector that efficiently identifies multimodal malicious attacks as a plug-and-play solution. Extensive experiments demonstrate superior performance across multiple dimensions, including robustness, efficiency, and utility. Through our work, we hope to pave the way for more secure multimodal AI deployment. Code is available at [this https URL](https://github.com/pgqihere/VLMShield). |
| title | VLMShield: Efficient and Robust Defense of Vision-Language Models against Malicious Prompts |
| topic | Machine Learning I.2; K.7 |
| url | https://arxiv.org/abs/2604.06502 |