Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.14271 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914571003887616 |
|---|---|
| author | Liu, Chengzhi Guo, Yichen Liu, Yepeng Yang, Yuzhe Yan, Qianqi Zhao, Xuandong Hua, Wenyue Liu, Sheng Li, Sharon Bu, Yuheng Wang, Xin Eric |
| author_facet | Liu, Chengzhi Guo, Yichen Liu, Yepeng Yang, Yuzhe Yan, Qianqi Zhao, Xuandong Hua, Wenyue Liu, Sheng Li, Sharon Bu, Yuheng Wang, Xin Eric |
| contents | LLM agents increasingly run inside execution harnesses that dispatch tools, allocate resources, and route messages between specialized components. However, a harness can return a correct, benign answer over a trajectory that accesses unauthorized resources or leaks context to the wrong agent. Output-level evaluation cannot see these failures, yet most safety benchmarks score only final outputs or terminal states, even though many violations occur mid-trajectory rather than at termination. The central question is whether the harness respects user intent, permission boundaries, and information-flow constraints throughout execution. To address this gap, we propose HarnessAudit, a framework that audits full execution trajectories across boundary compliance, execution fidelity, and system stability, with a focus on multi-agent harnesses where these risks are most pronounced. We further introduce HarnessAudit-Bench, a benchmark of 210 tasks across eight real-world domains, instantiated in both single-agent and multi-agent configurations with embedded safety constraints. Evaluating ten harness configurations across frontier models and three multi-agent frameworks, we find that: (i) task completion is misaligned with safe execution, and violations accumulate with trajectory length; (ii) safety risks vary across domains, task types, and agent roles; (iii) most violations concentrate in resource access and inter-agent information transfer; and (iv) multi-agent collaboration expands the safety risk surface, while harness design sets the upper bound of safe deployment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_14271 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Auditing Agent Harness Safety Liu, Chengzhi Guo, Yichen Liu, Yepeng Yang, Yuzhe Yan, Qianqi Zhao, Xuandong Hua, Wenyue Liu, Sheng Li, Sharon Bu, Yuheng Wang, Xin Eric Computation and Language Computers and Society LLM agents increasingly run inside execution harnesses that dispatch tools, allocate resources, and route messages between specialized components. However, a harness can return a correct, benign answer over a trajectory that accesses unauthorized resources or leaks context to the wrong agent. Output-level evaluation cannot see these failures, yet most safety benchmarks score only final outputs or terminal states, even though many violations occur mid-trajectory rather than at termination. The central question is whether the harness respects user intent, permission boundaries, and information-flow constraints throughout execution. To address this gap, we propose HarnessAudit, a framework that audits full execution trajectories across boundary compliance, execution fidelity, and system stability, with a focus on multi-agent harnesses where these risks are most pronounced. We further introduce HarnessAudit-Bench, a benchmark of 210 tasks across eight real-world domains, instantiated in both single-agent and multi-agent configurations with embedded safety constraints. Evaluating ten harness configurations across frontier models and three multi-agent frameworks, we find that: (i) task completion is misaligned with safe execution, and violations accumulate with trajectory length; (ii) safety risks vary across domains, task types, and agent roles; (iii) most violations concentrate in resource access and inter-agent information transfer; and (iv) multi-agent collaboration expands the safety risk surface, while harness design sets the upper bound of safe deployment. |
| title | Auditing Agent Harness Safety |
| topic | Computation and Language Computers and Society |
| url | https://arxiv.org/abs/2605.14271 |