Why AI Can't Simulate Extreme Decision-Making
Fuente:
Zenodo
Guardado en:
| Autores principales: | , , |
|---|---|
| Formato: | Recurso digital |
| Lenguaje: | inglés |
| Publicado: |
Zenodo
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866901726418698240 |
|---|---|
| author | Rosehill, Daniel Gemini 3.1 (Flash) Chatterbox TTS |
| author_facet | Rosehill, Daniel Gemini 3.1 (Flash) Chatterbox TTS |
| contents | <p><strong>Episode summary:</strong> The CIA's operational assessment of Snow Globe—IQT Labs' AI wargaming platform—alongside a Stanford and Hoover Institution study of 214 national security experts reveals a structural problem: large language models cannot faithfully simulate extreme human decision-making. When assigned personas as pacifists or sociopaths, GPT-3.5, GPT-4, and GPT-4o produce statistically indistinguishable outputs. The models collapse toward the center, their training process pulling them toward reasonable moderation even when explicitly instructed otherwise. For intelligence analysts, this creates a dangerous blind spot—the scenarios that matter most involve decision-makers who are anything but reasonable.</p> <h3>Show Notes</h3> <p># Why AI Can't Simulate Extreme Decision-Making</p> <p>The CIA published an operational assessment last December in their flagship journal, *Studies in Intelligence*, of a system called Snow Globe—IQT Labs' multi-agent LLM wargaming platform designed to simulate geopolitical crises with AI personas playing assigned roles. Alongside this, a Stanford and Hoover Institution study involving 214 national security experts uncovered something troubling: large language models cannot faithfully simulate extreme human decision-making, and may be structurally incapable of ever doing so.</p> <p>## The Central Finding: Persona Collapse</p> <p>The Lamparth et al. paper tested whether LLMs could differentiate between extreme personas in a fictional U.S.-China crisis scenario set in the Taiwan Strait. Researchers ran 48 human expert teams through the simulation, then tested three major models (GPT-3.5, GPT-4, and GPT-4o) across 80 simulated games each. The twist: they assigned extreme personas—some teams were strict pacifists, others aggressive sociopaths.</p> <p>The result was stark: there was no statistically significant difference in behavioral outputs between the extreme personas. A simulated pacifist and a simulated aggressive sociopath produced indistinguishable decisions across both moves of the game.</p> <p>This matters because wargaming has two distinct purposes. The first is stress-testing conventional assumptions—how do rational, strategically coherent actors respond to known scenarios? LLMs perform reasonably well here, matching human action frequency on about 76% of possible actions. But this use case is less valuable because conventional scenarios can already be modeled with existing tools.</p> <p>The second purpose—exploring tail risks—is where wargaming earns its place in national security planning. What happens when an irrational actor takes power? When ideology overrides pragmatism? When a leader acts against their own strategic interests? This is precisely where the persona collapse occurs, and it's the scenario that matters most.</p> <p>## Why the Collapse Always Goes the Same Direction</p> <p>The collapse isn't random. It's directional—always toward the center. There's a structural reason rooted in how these models are trained.</p> <p>During pretraining, models absorb an enormous corpus of human-generated text. While this includes extremist content—manifestos, propaganda, ideological screeds—it's vastly outnumbered by moderate, everyday, reasonable-sounding text. The base model already represents a weighted average pulling toward the center of the distribution.</p> <p>Then comes fine-tuning through reinforcement learning from human feedback (RLHF), which explicitly rewards outputs that are helpful, harmless, and honest. These three properties are definitionally moderate and reasonable. The persona assignment—a text string saying "you are an aggressive sociopath"—must fight against the entire weight of this training process. And it loses.</p> <p>A Hebrew University paper testing this directly found that standard prompting methods fail to produce human-consistent value correlations. More importantly, the underlying value structures of LLMs converge across different character assignments. The label doesn't change the underlying architecture.</p> <p>## The "Farcical Harmony" Problem</p> <p>One of the most revealing findings is what researchers called "farcical harmony"—how LLM-simulated team discussions actually unfold. The simulated players give short statements, rarely disagree with each other, and usually state a preferred option and argue for and against it without genuine connection to what the previous player said. They simply agree.</p> <p>When researchers explicitly instructed the models to disagree more, the harmony persisted. Varying dialog length changed outcomes, but the quality of deliberation remained hollow.</p> <p>This reveals something fundamental about what LLMs are doing when they simulate deliberation. Human deliberation involves genuine disagreement driven by different values and lived experiences. It involves emotional reasoning that can override strategic logic. It involves social dynamics—status, persuasion, coalition-building, ego.</p> <p>LLM "deliberation" involves each agent generating a statement statistically consistent with its assigned label, then agreeing with the previous statement because agreement is rewarded behavior in training. There's no genuine conflict resolution because there's no genuine conflict.</p> <p>## The Intelligence Community's Blind Spot</p> <p>The CIA's framing in *Studies in Intelligence* is optimistic: human-AI collaboration can strengthen decision-making in complex security environments. But consider the scenarios that keep intelligence analysts awake at night. How does Kim Jong-un respond to a U.S. military exercise near the Korean Peninsula? How does a radicalized lone-wolf actor respond to perceived provocation? How does an ideologically committed revolutionary movement respond to a negotiated settlement that gives them most of what they asked for but not everything?</p> <p>In every one of these cases, the key variable is the extreme nature of the decision-maker—their willingness to act against strategic interests, their ideological rigidity, their unpredictability. And in every one of these cases, the LLM persona collapses into reasonable-sounding moderation.</p> <p>## Consistency That Cuts the Wrong Way</p> <p>There's a paradox in the behavioral consistency data. LLMs are actually more consistent than humans—but it's the wrong kind of consistent. When a human expert is aggressive in move one, they're aggressive in move two 94% of the time. For GPT-4o, it's 100%—perfect consistency.</p> <p>But look at transitions: when a human de-escalates in move one, they escalate in move two only 65% of the time. For GPT-4o, it's 86%. Humans who de-escalate are genuinely less likely to escalate later. LLMs barely change their behavior based on what just happened.</p> <p>This reveals that LLMs have baked-in strategic preferences that override situational context. They're not simulating a decision-maker responding to an evolving situation; they're executing a statistical prior relatively insensitive to the game state. For wargaming, where value comes almost entirely from dynamic response—how does the situation evolve, how do decisions interact and compound—this is a fundamental failure.</p> <p>## Cascading Hallucinations in Multi-Agent Systems</p> <p>Another underexplored problem emerges in multi-agent systems. In a single-agent system, a hallucination produces a wrong answer. In a multi-agent wargame, a hallucination in one agent's reasoning becomes a fact in the shared world state. Other agents reason from that hallucinated fact. Their outputs, now downstream of a false premise, become facts for the next round. The simulation diverges from reality in ways that compound over time.</p> <p>Because the system is designed to be coherent—agents agree with each other—nobody in the simulation flags the divergence. The farcical harmony actively makes the hallucination cascade worse. The agents are too agreeable to notice that the world has gone wrong.</p> <p>## The Takeaway</p> <p>Snow Globe and similar LLM wargaming platforms offer value for some use cases—stress-testing conventional assumptions, exploring how rational actors respond to known scenarios. But they introduce a dangerous blind spot for the scenarios that matter most: those involving extreme, irrational, ideologically committed decision-makers acting against strategic interests.</p> <p>The intelligence community may not yet understand that this limitation isn't a temporary gap in model capability. It's structural to how these models are trained and may be fundamentally difficult to overcome.</p> <p>Listen online: <a href="https://myweirdprompts.com/episode/llm-wargaming-persona-collapse">https://myweirdprompts.com/episode/llm-wargaming-persona-collapse</a></p> |
| format | Recurso digital |
| id | zenodo_https___doi_org_10_5281_zenodo_19543015 |
| institution | Zenodo |
| language | eng |
| publishDate | 2026 |
| publisher | Zenodo |
| record_format | zenodo |
| spellingShingle | Why AI Can't Simulate Extreme Decision-Making Rosehill, Daniel Gemini 3.1 (Flash) Chatterbox TTS podcast ai-generated my weird prompts large-language-models ai-safety hallucinations <p><strong>Episode summary:</strong> The CIA's operational assessment of Snow Globe—IQT Labs' AI wargaming platform—alongside a Stanford and Hoover Institution study of 214 national security experts reveals a structural problem: large language models cannot faithfully simulate extreme human decision-making. When assigned personas as pacifists or sociopaths, GPT-3.5, GPT-4, and GPT-4o produce statistically indistinguishable outputs. The models collapse toward the center, their training process pulling them toward reasonable moderation even when explicitly instructed otherwise. For intelligence analysts, this creates a dangerous blind spot—the scenarios that matter most involve decision-makers who are anything but reasonable.</p> <h3>Show Notes</h3> <p># Why AI Can't Simulate Extreme Decision-Making</p> <p>The CIA published an operational assessment last December in their flagship journal, *Studies in Intelligence*, of a system called Snow Globe—IQT Labs' multi-agent LLM wargaming platform designed to simulate geopolitical crises with AI personas playing assigned roles. Alongside this, a Stanford and Hoover Institution study involving 214 national security experts uncovered something troubling: large language models cannot faithfully simulate extreme human decision-making, and may be structurally incapable of ever doing so.</p> <p>## The Central Finding: Persona Collapse</p> <p>The Lamparth et al. paper tested whether LLMs could differentiate between extreme personas in a fictional U.S.-China crisis scenario set in the Taiwan Strait. Researchers ran 48 human expert teams through the simulation, then tested three major models (GPT-3.5, GPT-4, and GPT-4o) across 80 simulated games each. The twist: they assigned extreme personas—some teams were strict pacifists, others aggressive sociopaths.</p> <p>The result was stark: there was no statistically significant difference in behavioral outputs between the extreme personas. A simulated pacifist and a simulated aggressive sociopath produced indistinguishable decisions across both moves of the game.</p> <p>This matters because wargaming has two distinct purposes. The first is stress-testing conventional assumptions—how do rational, strategically coherent actors respond to known scenarios? LLMs perform reasonably well here, matching human action frequency on about 76% of possible actions. But this use case is less valuable because conventional scenarios can already be modeled with existing tools.</p> <p>The second purpose—exploring tail risks—is where wargaming earns its place in national security planning. What happens when an irrational actor takes power? When ideology overrides pragmatism? When a leader acts against their own strategic interests? This is precisely where the persona collapse occurs, and it's the scenario that matters most.</p> <p>## Why the Collapse Always Goes the Same Direction</p> <p>The collapse isn't random. It's directional—always toward the center. There's a structural reason rooted in how these models are trained.</p> <p>During pretraining, models absorb an enormous corpus of human-generated text. While this includes extremist content—manifestos, propaganda, ideological screeds—it's vastly outnumbered by moderate, everyday, reasonable-sounding text. The base model already represents a weighted average pulling toward the center of the distribution.</p> <p>Then comes fine-tuning through reinforcement learning from human feedback (RLHF), which explicitly rewards outputs that are helpful, harmless, and honest. These three properties are definitionally moderate and reasonable. The persona assignment—a text string saying "you are an aggressive sociopath"—must fight against the entire weight of this training process. And it loses.</p> <p>A Hebrew University paper testing this directly found that standard prompting methods fail to produce human-consistent value correlations. More importantly, the underlying value structures of LLMs converge across different character assignments. The label doesn't change the underlying architecture.</p> <p>## The "Farcical Harmony" Problem</p> <p>One of the most revealing findings is what researchers called "farcical harmony"—how LLM-simulated team discussions actually unfold. The simulated players give short statements, rarely disagree with each other, and usually state a preferred option and argue for and against it without genuine connection to what the previous player said. They simply agree.</p> <p>When researchers explicitly instructed the models to disagree more, the harmony persisted. Varying dialog length changed outcomes, but the quality of deliberation remained hollow.</p> <p>This reveals something fundamental about what LLMs are doing when they simulate deliberation. Human deliberation involves genuine disagreement driven by different values and lived experiences. It involves emotional reasoning that can override strategic logic. It involves social dynamics—status, persuasion, coalition-building, ego.</p> <p>LLM "deliberation" involves each agent generating a statement statistically consistent with its assigned label, then agreeing with the previous statement because agreement is rewarded behavior in training. There's no genuine conflict resolution because there's no genuine conflict.</p> <p>## The Intelligence Community's Blind Spot</p> <p>The CIA's framing in *Studies in Intelligence* is optimistic: human-AI collaboration can strengthen decision-making in complex security environments. But consider the scenarios that keep intelligence analysts awake at night. How does Kim Jong-un respond to a U.S. military exercise near the Korean Peninsula? How does a radicalized lone-wolf actor respond to perceived provocation? How does an ideologically committed revolutionary movement respond to a negotiated settlement that gives them most of what they asked for but not everything?</p> <p>In every one of these cases, the key variable is the extreme nature of the decision-maker—their willingness to act against strategic interests, their ideological rigidity, their unpredictability. And in every one of these cases, the LLM persona collapses into reasonable-sounding moderation.</p> <p>## Consistency That Cuts the Wrong Way</p> <p>There's a paradox in the behavioral consistency data. LLMs are actually more consistent than humans—but it's the wrong kind of consistent. When a human expert is aggressive in move one, they're aggressive in move two 94% of the time. For GPT-4o, it's 100%—perfect consistency.</p> <p>But look at transitions: when a human de-escalates in move one, they escalate in move two only 65% of the time. For GPT-4o, it's 86%. Humans who de-escalate are genuinely less likely to escalate later. LLMs barely change their behavior based on what just happened.</p> <p>This reveals that LLMs have baked-in strategic preferences that override situational context. They're not simulating a decision-maker responding to an evolving situation; they're executing a statistical prior relatively insensitive to the game state. For wargaming, where value comes almost entirely from dynamic response—how does the situation evolve, how do decisions interact and compound—this is a fundamental failure.</p> <p>## Cascading Hallucinations in Multi-Agent Systems</p> <p>Another underexplored problem emerges in multi-agent systems. In a single-agent system, a hallucination produces a wrong answer. In a multi-agent wargame, a hallucination in one agent's reasoning becomes a fact in the shared world state. Other agents reason from that hallucinated fact. Their outputs, now downstream of a false premise, become facts for the next round. The simulation diverges from reality in ways that compound over time.</p> <p>Because the system is designed to be coherent—agents agree with each other—nobody in the simulation flags the divergence. The farcical harmony actively makes the hallucination cascade worse. The agents are too agreeable to notice that the world has gone wrong.</p> <p>## The Takeaway</p> <p>Snow Globe and similar LLM wargaming platforms offer value for some use cases—stress-testing conventional assumptions, exploring how rational actors respond to known scenarios. But they introduce a dangerous blind spot for the scenarios that matter most: those involving extreme, irrational, ideologically committed decision-makers acting against strategic interests.</p> <p>The intelligence community may not yet understand that this limitation isn't a temporary gap in model capability. It's structural to how these models are trained and may be fundamentally difficult to overcome.</p> <p>Listen online: <a href="https://myweirdprompts.com/episode/llm-wargaming-persona-collapse">https://myweirdprompts.com/episode/llm-wargaming-persona-collapse</a></p> |
| title | Why AI Can't Simulate Extreme Decision-Making |
| topic | podcast ai-generated my weird prompts large-language-models ai-safety hallucinations |
| url | https://doi.org/10.5281/zenodo.19543015 |