Optimizing Agent Planning for Security and Autonomy
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910019913515008 |
|---|---|
| author | Kolluri, Aashish Sharma, Rishi Costa, Manuel Köpf, Boris Nießen, Tobias Russinovich, Mark Tople, Shruti Zanella-Béguelin, Santiago |
| author_facet | Kolluri, Aashish Sharma, Rishi Costa, Manuel Köpf, Boris Nießen, Tobias Russinovich, Mark Tople, Shruti Zanella-Béguelin, Santiago |
| contents | Indirect prompt injection attacks threaten AI agents that execute consequential actions, motivating deterministic system-level defenses. Such defenses can provably block unsafe actions by enforcing confidentiality and integrity policies, but currently appear costly: they reduce task completion rates and increase token usage compared to probabilistic defenses. We argue that existing evaluations miss a key benefit of system-level defenses: reduced reliance on human oversight. We introduce autonomy metrics to quantify this benefit: the fraction of consequential actions an agent can execute without human-in-the-loop (HITL) approval while preserving security. To increase autonomy, we design a security-aware agent that (i) introduces richer HITL interactions, and (ii) explicitly plans for both task progress and policy compliance. We implement this agent design atop an existing information-flow control defense against prompt injection and evaluate it on the AgentDojo and WASP benchmarks. Experiments show that this approach yields higher autonomy without sacrificing utility. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_11416 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Optimizing Agent Planning for Security and Autonomy Kolluri, Aashish Sharma, Rishi Costa, Manuel Köpf, Boris Nießen, Tobias Russinovich, Mark Tople, Shruti Zanella-Béguelin, Santiago Cryptography and Security Machine Learning I.2.8; D.4.6 Indirect prompt injection attacks threaten AI agents that execute consequential actions, motivating deterministic system-level defenses. Such defenses can provably block unsafe actions by enforcing confidentiality and integrity policies, but currently appear costly: they reduce task completion rates and increase token usage compared to probabilistic defenses. We argue that existing evaluations miss a key benefit of system-level defenses: reduced reliance on human oversight. We introduce autonomy metrics to quantify this benefit: the fraction of consequential actions an agent can execute without human-in-the-loop (HITL) approval while preserving security. To increase autonomy, we design a security-aware agent that (i) introduces richer HITL interactions, and (ii) explicitly plans for both task progress and policy compliance. We implement this agent design atop an existing information-flow control defense against prompt injection and evaluate it on the AgentDojo and WASP benchmarks. Experiments show that this approach yields higher autonomy without sacrificing utility. |
| title | Optimizing Agent Planning for Security and Autonomy |
| topic | Cryptography and Security Machine Learning I.2.8; D.4.6 |
| url | https://arxiv.org/abs/2602.11416 |