Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2405.01870 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908863531319296 |
|---|---|
| author | Alon, Nitay Barnby, Joseph M. Sarkadi, Stefan Schulz, Lion Rosenschein, Jeffrey S. Dayan, Peter |
| author_facet | Alon, Nitay Barnby, Joseph M. Sarkadi, Stefan Schulz, Lion Rosenschein, Jeffrey S. Dayan, Peter |
| contents | Social agents with finitely nested opponent models are vulnerable to manipulation by agents with deeper recursive capabilities. This imbalance, rooted in logic and the theory of recursive modelling frameworks, cannot be solved directly. We propose a computational framework called $\aleph$-IPOMDP, which augments the Bayesian inference of model-based RL agents with an anomaly detection algorithm and an out-of-belief policy. Our mechanism allows agents to realize that they are being deceived, even if they cannot understand how, and to deter opponents via a credible threat. We test this framework in both a mixed-motive and a zero-sum game. Our results demonstrate the $\aleph$-mechanism's effectiveness, leading to more equitable outcomes and less exploitation by more sophisticated agents. We discuss implications for AI safety, cybersecurity, cognitive science, and psychiatry. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_01870 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | $\aleph$-IPOMDP: Mitigating Deception in a Cognitive Hierarchy with Off-Policy Counterfactual Anomaly Detection Alon, Nitay Barnby, Joseph M. Sarkadi, Stefan Schulz, Lion Rosenschein, Jeffrey S. Dayan, Peter Multiagent Systems Computer Science and Game Theory Social agents with finitely nested opponent models are vulnerable to manipulation by agents with deeper recursive capabilities. This imbalance, rooted in logic and the theory of recursive modelling frameworks, cannot be solved directly. We propose a computational framework called $\aleph$-IPOMDP, which augments the Bayesian inference of model-based RL agents with an anomaly detection algorithm and an out-of-belief policy. Our mechanism allows agents to realize that they are being deceived, even if they cannot understand how, and to deter opponents via a credible threat. We test this framework in both a mixed-motive and a zero-sum game. Our results demonstrate the $\aleph$-mechanism's effectiveness, leading to more equitable outcomes and less exploitation by more sophisticated agents. We discuss implications for AI safety, cybersecurity, cognitive science, and psychiatry. |
| title | $\aleph$-IPOMDP: Mitigating Deception in a Cognitive Hierarchy with Off-Policy Counterfactual Anomaly Detection |
| topic | Multiagent Systems Computer Science and Game Theory |
| url | https://arxiv.org/abs/2405.01870 |