Emergent Risk Awareness in Rational Agents under Resource Constraints
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918146836791296 |
|---|---|
| author | Ornia, Daniel Jarne Bishop, Nicholas Dyer, Joel Lee, Wei-Chen Calinescu, Ani Farmer, Doyne Wooldridge, Michael |
| author_facet | Ornia, Daniel Jarne Bishop, Nicholas Dyer, Joel Lee, Wei-Chen Calinescu, Ani Farmer, Doyne Wooldridge, Michael |
| contents | Advanced reasoning models with agentic capabilities (AI agents) are deployed to interact with humans and to solve sequential decision-making problems under (approximate) utility functions and internal models. When such problems have resource or failure constraints where action sequences may be forcibly terminated once resources are exhausted, agents face implicit trade-offs that reshape their utility-driven (rational) behaviour. Additionally, since these agents are typically commissioned by a human principal to act on their behalf, asymmetries in constraint exposure can give rise to previously unanticipated misalignment between human objectives and agent incentives. We formalise this setting through a survival bandit framework, provide theoretical and empirical results that quantify the impact of survival-driven preference shifts, identify conditions under which misalignment emerges and propose mechanisms to mitigate the emergence of risk-seeking or risk-averse behaviours. As a result, this work aims to increase understanding and interpretability of emergent behaviours of AI agents operating under such survival pressure, and offer guidelines for safely deploying such AI systems in critical resource-limited environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_23436 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Emergent Risk Awareness in Rational Agents under Resource Constraints Ornia, Daniel Jarne Bishop, Nicholas Dyer, Joel Lee, Wei-Chen Calinescu, Ani Farmer, Doyne Wooldridge, Michael Artificial Intelligence Machine Learning Advanced reasoning models with agentic capabilities (AI agents) are deployed to interact with humans and to solve sequential decision-making problems under (approximate) utility functions and internal models. When such problems have resource or failure constraints where action sequences may be forcibly terminated once resources are exhausted, agents face implicit trade-offs that reshape their utility-driven (rational) behaviour. Additionally, since these agents are typically commissioned by a human principal to act on their behalf, asymmetries in constraint exposure can give rise to previously unanticipated misalignment between human objectives and agent incentives. We formalise this setting through a survival bandit framework, provide theoretical and empirical results that quantify the impact of survival-driven preference shifts, identify conditions under which misalignment emerges and propose mechanisms to mitigate the emergence of risk-seeking or risk-averse behaviours. As a result, this work aims to increase understanding and interpretability of emergent behaviours of AI agents operating under such survival pressure, and offer guidelines for safely deploying such AI systems in critical resource-limited environments. |
| title | Emergent Risk Awareness in Rational Agents under Resource Constraints |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2505.23436 |