SeClaw: Spec-Driven Security Task Synthesis for Evaluating Autonomous Agents
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910281987260416 |
|---|---|
| author | Cheng, Hao Miao, Changtao Song, Tianle Wu, Yin Liu, He Xiao, Erjia Chen, Junchi Shi, Xiaoyu Wang, Yichi Yang, Jing Wang, Taowen Duan, Jinhao Sun, Mengshu Dong, Peiyan Shen, Xuan Cao, Yang Xu, Renjing Xu, Kaidi Gu, Jindong Zhang, Bo Zhang, Jize Lin, Chenhao Torr, Philip Shen, Chao |
| author_facet | Cheng, Hao Miao, Changtao Song, Tianle Wu, Yin Liu, He Xiao, Erjia Chen, Junchi Shi, Xiaoyu Wang, Yichi Yang, Jing Wang, Taowen Duan, Jinhao Sun, Mengshu Dong, Peiyan Shen, Xuan Cao, Yang Xu, Renjing Xu, Kaidi Gu, Jindong Zhang, Bo Zhang, Jize Lin, Chenhao Torr, Philip Shen, Chao |
| contents | Autonomous LLM agents increasingly operate in stateful environments where they access tools, files, memory, and external services. While such capabilities enable complex real-world workflows, they also introduce security risks that are difficult to capture with existing evaluations. Current agent security benchmarks often rely on manually curated tasks, provide limited coverage of emerging threats, and focus primarily on final outcomes rather than the execution processes that lead to unsafe behavior. We introduce SeClaw, a framework that combines specification-driven security task synthesis with execution-based security evaluation for Autonomous agents. Spec-driven security task synthesis enables scalable and controllable construction of security tasks from structured risk specifications, while SeClaw docker provides a standardized testbed for evaluating agent behavior under diverse safety-risk scenarios. The benchmark covers risks arising from resources, user tasks, environments, and intrinsic agent behaviors, and supports trajectory-aware assessment of unsafe actions beyond final responses. By bridging systematic task synthesis and reproducible security evaluation, SeClaw provides a practical foundation for measuring, diagnosing, and comparing security failures in autonomous LLM agents. The code is available at https://github.com/seclaw-eval/seclaw-eval. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2606_02302 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SeClaw: Spec-Driven Security Task Synthesis for Evaluating Autonomous Agents Cheng, Hao Miao, Changtao Song, Tianle Wu, Yin Liu, He Xiao, Erjia Chen, Junchi Shi, Xiaoyu Wang, Yichi Yang, Jing Wang, Taowen Duan, Jinhao Sun, Mengshu Dong, Peiyan Shen, Xuan Cao, Yang Xu, Renjing Xu, Kaidi Gu, Jindong Zhang, Bo Zhang, Jize Lin, Chenhao Torr, Philip Shen, Chao Cryptography and Security Artificial Intelligence Autonomous LLM agents increasingly operate in stateful environments where they access tools, files, memory, and external services. While such capabilities enable complex real-world workflows, they also introduce security risks that are difficult to capture with existing evaluations. Current agent security benchmarks often rely on manually curated tasks, provide limited coverage of emerging threats, and focus primarily on final outcomes rather than the execution processes that lead to unsafe behavior. We introduce SeClaw, a framework that combines specification-driven security task synthesis with execution-based security evaluation for Autonomous agents. Spec-driven security task synthesis enables scalable and controllable construction of security tasks from structured risk specifications, while SeClaw docker provides a standardized testbed for evaluating agent behavior under diverse safety-risk scenarios. The benchmark covers risks arising from resources, user tasks, environments, and intrinsic agent behaviors, and supports trajectory-aware assessment of unsafe actions beyond final responses. By bridging systematic task synthesis and reproducible security evaluation, SeClaw provides a practical foundation for measuring, diagnosing, and comparing security failures in autonomous LLM agents. The code is available at https://github.com/seclaw-eval/seclaw-eval. |
| title | SeClaw: Spec-Driven Security Task Synthesis for Evaluating Autonomous Agents |
| topic | Cryptography and Security Artificial Intelligence |
| url | https://arxiv.org/abs/2606.02302 |