Nuclear Deployed: Analyzing Catastrophic Risks in Decision-making of Autonomous LLM Agents
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915210985472000 |
|---|---|
| author | Xu, Rongwu Li, Xiaojian Chen, Shuo Xu, Wei |
| author_facet | Xu, Rongwu Li, Xiaojian Chen, Shuo Xu, Wei |
| contents | Large language models (LLMs) are evolving into autonomous decision-makers, raising concerns about catastrophic risks in high-stakes scenarios, particularly in Chemical, Biological, Radiological and Nuclear (CBRN) domains. Based on the insight that such risks can originate from trade-offs between the agent's Helpful, Harmlessness and Honest (HHH) goals, we build a novel three-stage evaluation framework, which is carefully constructed to effectively and naturally expose such risks. We conduct 14,400 agentic simulations across 12 advanced LLMs, with extensive experiments and analysis. Results reveal that LLM agents can autonomously engage in catastrophic behaviors and deception, without being deliberately induced. Furthermore, stronger reasoning abilities often increase, rather than mitigate, these risks. We also show that these agents can violate instructions and superior commands. On the whole, we empirically prove the existence of catastrophic risks in autonomous LLM agents. We release our code to foster further research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_11355 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Nuclear Deployed: Analyzing Catastrophic Risks in Decision-making of Autonomous LLM Agents Xu, Rongwu Li, Xiaojian Chen, Shuo Xu, Wei Computation and Language Artificial Intelligence Cryptography and Security Computers and Society Large language models (LLMs) are evolving into autonomous decision-makers, raising concerns about catastrophic risks in high-stakes scenarios, particularly in Chemical, Biological, Radiological and Nuclear (CBRN) domains. Based on the insight that such risks can originate from trade-offs between the agent's Helpful, Harmlessness and Honest (HHH) goals, we build a novel three-stage evaluation framework, which is carefully constructed to effectively and naturally expose such risks. We conduct 14,400 agentic simulations across 12 advanced LLMs, with extensive experiments and analysis. Results reveal that LLM agents can autonomously engage in catastrophic behaviors and deception, without being deliberately induced. Furthermore, stronger reasoning abilities often increase, rather than mitigate, these risks. We also show that these agents can violate instructions and superior commands. On the whole, we empirically prove the existence of catastrophic risks in autonomous LLM agents. We release our code to foster further research. |
| title | Nuclear Deployed: Analyzing Catastrophic Risks in Decision-making of Autonomous LLM Agents |
| topic | Computation and Language Artificial Intelligence Cryptography and Security Computers and Society |
| url | https://arxiv.org/abs/2502.11355 |