SkillSafetyBench: Evaluating Agent Safety under Skill-Facing Attack Surfaces
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911722468540416 |
|---|---|
| author | Jin, Chang Wang, An Wei, Zeming Wang, Kai Zeng, Biaojie Zhang, Qiaosheng Yang, Chao Qu, Jingjing Hu, Xia Xu, Xingcheng |
| author_facet | Jin, Chang Wang, An Wei, Zeming Wang, Kai Zeng, Biaojie Zhang, Qiaosheng Yang, Chao Qu, Jingjing Hu, Xia Xu, Xingcheng |
| contents | Reusable skills are becoming a common interface for extending large language model agents, packaging procedural guidance with access to files, tools, memory, and execution environments. However, this modularity introduces attack surfaces that are largely missed by existing safety evaluations: even when the user request is benign, unsafe influence may reside in skill guidance, local artifacts, or execution-environment files that steer the agent toward unsafe actions. We present SkillSafetyBench, a runnable benchmark for evaluating such skill-mediated safety failures. SkillSafetyBench includes 155 adversarial cases across 47 tasks, 6 risk domains, and 30 safety categories, each evaluated with a case-specific rule-based verifier. Experiments with multiple CLI agents and model backends show that non-user attacks can consistently induce unsafe behavior, with distinct failure patterns across domains, attack methods, and scaffold-model pairings. Our findings suggest that agent safety depends not only on model-level alignment, but also on how agents interpret skills, trust workflow context, and act through executable environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_12015 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SkillSafetyBench: Evaluating Agent Safety under Skill-Facing Attack Surfaces Jin, Chang Wang, An Wei, Zeming Wang, Kai Zeng, Biaojie Zhang, Qiaosheng Yang, Chao Qu, Jingjing Hu, Xia Xu, Xingcheng Cryptography and Security Artificial Intelligence Computation and Language Machine Learning Multiagent Systems Reusable skills are becoming a common interface for extending large language model agents, packaging procedural guidance with access to files, tools, memory, and execution environments. However, this modularity introduces attack surfaces that are largely missed by existing safety evaluations: even when the user request is benign, unsafe influence may reside in skill guidance, local artifacts, or execution-environment files that steer the agent toward unsafe actions. We present SkillSafetyBench, a runnable benchmark for evaluating such skill-mediated safety failures. SkillSafetyBench includes 155 adversarial cases across 47 tasks, 6 risk domains, and 30 safety categories, each evaluated with a case-specific rule-based verifier. Experiments with multiple CLI agents and model backends show that non-user attacks can consistently induce unsafe behavior, with distinct failure patterns across domains, attack methods, and scaffold-model pairings. Our findings suggest that agent safety depends not only on model-level alignment, but also on how agents interpret skills, trust workflow context, and act through executable environments. |
| title | SkillSafetyBench: Evaluating Agent Safety under Skill-Facing Attack Surfaces |
| topic | Cryptography and Security Artificial Intelligence Computation and Language Machine Learning Multiagent Systems |
| url | https://arxiv.org/abs/2605.12015 |