Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2604.15384 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913066594074624 |
|---|---|
| author | Tracy, Tyler Potham, Ram Kuhn, Nick Heller, Myles Khandelwal, Anshul Rushing, Cody Lemoine, Henri Brandao, Miguel Turlik, Tomas Hanson, Adam Hills, Josh Ngo, Amy Rachum, Ram Mitchell, Nik Galperin, Falko Sykes, Oscar Arnott, Pip Lima, Samuel Prieto Giudice, Carlos Goldwater, Matt Popp, Daniel de Wet, Drew Castaing, Ruben Guo, Qi Marx, Douw Shaffrey, Benjamin Shenk, Justin Milbradt, Martin Meagher, Hannah Ahmed-Chowdhury, Shaheen O'Connell, Daniel Canal, Chris Shlegeris, Buck Bhatt, Aryan |
| author_facet | Tracy, Tyler Potham, Ram Kuhn, Nick Heller, Myles Khandelwal, Anshul Rushing, Cody Lemoine, Henri Brandao, Miguel Turlik, Tomas Hanson, Adam Hills, Josh Ngo, Amy Rachum, Ram Mitchell, Nik Galperin, Falko Sykes, Oscar Arnott, Pip Lima, Samuel Prieto Giudice, Carlos Goldwater, Matt Popp, Daniel de Wet, Drew Castaing, Ruben Guo, Qi Marx, Douw Shaffrey, Benjamin Shenk, Justin Milbradt, Martin Meagher, Hannah Ahmed-Chowdhury, Shaheen O'Connell, Daniel Canal, Chris Shlegeris, Buck Bhatt, Aryan |
| contents | We introduce LinuxArena, a control setting in which agents operate directly on live, multi-service production environments. LinuxArena contains 20 environments, 1,671 main tasks representing legitimate software engineering work, and 184 side tasks representing safety failures such as data exfiltration and backdooring, making it the largest and most diverse control setting for software engineering to date. We validate LinuxArena is useful for control research by running sabotage evaluations, which measure whether attackers can complete side tasks while working on main tasks, and monitor evaluations, which measure a monitor model's ability to detect sabotage attempts. Against a GPT-5-nano trusted monitor at a 1\% step-wise false positive rate, Claude Opus 4.6 achieves roughly a 23% undetected sabotage success rate. We additionally release LaStraj, a dataset of human-crafted attack trajectories that evade monitors at substantially higher rates than any model-generated attacks we elicited, showing that current attack policies do not saturate LinuxArena. These results suggest that LinuxArena has meaningful headroom for both attackers and defenders, making it a strong testbed for developing and evaluating future control protocols. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_15384 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | LinuxArena: A Control Setting for AI Agents in Live Production Software Environments Tracy, Tyler Potham, Ram Kuhn, Nick Heller, Myles Khandelwal, Anshul Rushing, Cody Lemoine, Henri Brandao, Miguel Turlik, Tomas Hanson, Adam Hills, Josh Ngo, Amy Rachum, Ram Mitchell, Nik Galperin, Falko Sykes, Oscar Arnott, Pip Lima, Samuel Prieto Giudice, Carlos Goldwater, Matt Popp, Daniel de Wet, Drew Castaing, Ruben Guo, Qi Marx, Douw Shaffrey, Benjamin Shenk, Justin Milbradt, Martin Meagher, Hannah Ahmed-Chowdhury, Shaheen O'Connell, Daniel Canal, Chris Shlegeris, Buck Bhatt, Aryan Cryptography and Security Artificial Intelligence Software Engineering We introduce LinuxArena, a control setting in which agents operate directly on live, multi-service production environments. LinuxArena contains 20 environments, 1,671 main tasks representing legitimate software engineering work, and 184 side tasks representing safety failures such as data exfiltration and backdooring, making it the largest and most diverse control setting for software engineering to date. We validate LinuxArena is useful for control research by running sabotage evaluations, which measure whether attackers can complete side tasks while working on main tasks, and monitor evaluations, which measure a monitor model's ability to detect sabotage attempts. Against a GPT-5-nano trusted monitor at a 1\% step-wise false positive rate, Claude Opus 4.6 achieves roughly a 23% undetected sabotage success rate. We additionally release LaStraj, a dataset of human-crafted attack trajectories that evade monitors at substantially higher rates than any model-generated attacks we elicited, showing that current attack policies do not saturate LinuxArena. These results suggest that LinuxArena has meaningful headroom for both attackers and defenders, making it a strong testbed for developing and evaluating future control protocols. |
| title | LinuxArena: A Control Setting for AI Agents in Live Production Software Environments |
| topic | Cryptography and Security Artificial Intelligence Software Engineering |
| url | https://arxiv.org/abs/2604.15384 |