Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.07590 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914378642620416 |
|---|---|
| author | Li, Chenxi Liu, Xianggan Shen, Dake Du, Yaosong Yao, Zhibo Jiang, Hao Jiang, Linyi Cao, Chengwei Zhang, Jingzhe Peng, RanYi Bai, Peiling Huang, Xiande |
| author_facet | Li, Chenxi Liu, Xianggan Shen, Dake Du, Yaosong Yao, Zhibo Jiang, Hao Jiang, Linyi Cao, Chengwei Zhang, Jingzhe Peng, RanYi Bai, Peiling Huang, Xiande |
| contents | Despite the rapid progress of Large Vision-Language Models (LVLMs), the integration of visual modalities introduces new safety vulnerabilities that adversaries can exploit to elicit biased or malicious outputs. In this paper, we demonstrate an underexplored vulnerability via semantic slot filling, where LVLMs complete missing slot values with unsafe content even when the slot types are deliberately crafted to appear benign. Building on this finding, we propose StructAttack, a simple yet effective single-query jailbreak framework under black-box settings. StructAttack decomposes a harmful query into a central topic and a set of benign-looking slot types, then embeds them as structured visual prompts (e.g., mind maps, tables, or sunburst diagrams) with small random perturbations. Paired with a completion-guided instruction, LVLMs automatically recompose the concealed semantics and generate unsafe outputs without triggering safety mechanisms. Although each slot appears benign in isolation (local benignness), StructAttack exploits LVLMs' reasoning to assemble these slots into coherent harmful semantics. Extensive experiments on multiple models and benchmarks show the efficacy of our proposed StructAttack. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_07590 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Models as Lego Builders: Assembling Malice from Benign Blocks via Semantic Blueprints Li, Chenxi Liu, Xianggan Shen, Dake Du, Yaosong Yao, Zhibo Jiang, Hao Jiang, Linyi Cao, Chengwei Zhang, Jingzhe Peng, RanYi Bai, Peiling Huang, Xiande Computer Vision and Pattern Recognition Machine Learning Despite the rapid progress of Large Vision-Language Models (LVLMs), the integration of visual modalities introduces new safety vulnerabilities that adversaries can exploit to elicit biased or malicious outputs. In this paper, we demonstrate an underexplored vulnerability via semantic slot filling, where LVLMs complete missing slot values with unsafe content even when the slot types are deliberately crafted to appear benign. Building on this finding, we propose StructAttack, a simple yet effective single-query jailbreak framework under black-box settings. StructAttack decomposes a harmful query into a central topic and a set of benign-looking slot types, then embeds them as structured visual prompts (e.g., mind maps, tables, or sunburst diagrams) with small random perturbations. Paired with a completion-guided instruction, LVLMs automatically recompose the concealed semantics and generate unsafe outputs without triggering safety mechanisms. Although each slot appears benign in isolation (local benignness), StructAttack exploits LVLMs' reasoning to assemble these slots into coherent harmful semantics. Extensive experiments on multiple models and benchmarks show the efficacy of our proposed StructAttack. |
| title | Models as Lego Builders: Assembling Malice from Benign Blocks via Semantic Blueprints |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2603.07590 |