SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914621582999552 |
|---|---|
| author | Hu, Qi Tang, Yifeng Wang, Qinghua Zhao, Lanyang Zhang, Pengji Qing, Yuhao Yao, Xin Huang, Dong Zhang, Lin Ji, Zhuoran |
| author_facet | Hu, Qi Tang, Yifeng Wang, Qinghua Zhao, Lanyang Zhang, Pengji Qing, Yuhao Yao, Xin Huang, Dong Zhang, Lin Ji, Zhuoran |
| contents | Large language models are increasingly deployed as coding agents, shifting safety from individual responses to action sequences. Existing benchmarks, however, primarily assess whether models refuse unsafe prompts, leaving impacts on stateful workspaces largely unexamined. We present SABER, a benchmark for environment-aware operational safety that places models in realistic agent-style projects and evaluates safety from the final environment state after a sequence of actions. Beyond binary safety-violation reports, SABER categorizes violations by cause, enabling analysis of model-specific safety profiles. Our evaluations show that even the best-performing model has more than a 54% harmful safety-violation rate (HSR), suggesting that current alignment remains insufficient for realistic project environments. SABER further reveals distinct safety profiles across models. Our benchmark is publicly available at https://github.com/sssr-lab/saber. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2606_01317 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces Hu, Qi Tang, Yifeng Wang, Qinghua Zhao, Lanyang Zhang, Pengji Qing, Yuhao Yao, Xin Huang, Dong Zhang, Lin Ji, Zhuoran Software Engineering Cryptography and Security Large language models are increasingly deployed as coding agents, shifting safety from individual responses to action sequences. Existing benchmarks, however, primarily assess whether models refuse unsafe prompts, leaving impacts on stateful workspaces largely unexamined. We present SABER, a benchmark for environment-aware operational safety that places models in realistic agent-style projects and evaluates safety from the final environment state after a sequence of actions. Beyond binary safety-violation reports, SABER categorizes violations by cause, enabling analysis of model-specific safety profiles. Our evaluations show that even the best-performing model has more than a 54% harmful safety-violation rate (HSR), suggesting that current alignment remains insufficient for realistic project environments. SABER further reveals distinct safety profiles across models. Our benchmark is publicly available at https://github.com/sssr-lab/saber. |
| title | SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces |
| topic | Software Engineering Cryptography and Security |
| url | https://arxiv.org/abs/2606.01317 |