Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.07485 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909731360079872 |
|---|---|
| author | Duffy, Alexander Paech, Samuel J Shastri, Ishana Karpinski, Elizabeth Alloui-Cros, Baptiste Marques, Tyler Olson, Matthew Lyle |
| author_facet | Duffy, Alexander Paech, Samuel J Shastri, Ishana Karpinski, Elizabeth Alloui-Cros, Baptiste Marques, Tyler Olson, Matthew Lyle |
| contents | We present the first evaluation harness that enables any out-of-the-box, local, Large Language Models (LLMs) to play full-press Diplomacy without fine-tuning or specialized training. Previous work required frontier LLMs, or fine-tuning, due to the high complexity and information density of Diplomacy's game state. Combined with the high variance of matches, these factors made Diplomacy prohibitive for study. In this work, we used data-driven iteration to optimize a textual game state representation such that a 24B model can reliably complete matches without any fine tuning. We develop tooling to facilitate hypothesis testing and statistical analysis, and we present case studies on persuasion, aggressive playstyles, and performance across a range of models. We conduct a variety of experiments across many popular LLMs, finding the larger models perform the best, but the smaller models still play adequately. We also introduce Critical State Analysis: an experimental protocol for rapidly iterating and analyzing key moments in a game at depth. Our harness democratizes the evaluation of strategic reasoning in LLMs by eliminating the need for fine-tuning, and it provides insights into how these capabilities emerge naturally from widely used LLMs. Our code is available in the supplement and will be open sourced. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_07485 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Democratizing Diplomacy: A Harness for Evaluating Any Large Language Model on Full-Press Diplomacy Duffy, Alexander Paech, Samuel J Shastri, Ishana Karpinski, Elizabeth Alloui-Cros, Baptiste Marques, Tyler Olson, Matthew Lyle Artificial Intelligence Computation and Language Computers and Society Machine Learning We present the first evaluation harness that enables any out-of-the-box, local, Large Language Models (LLMs) to play full-press Diplomacy without fine-tuning or specialized training. Previous work required frontier LLMs, or fine-tuning, due to the high complexity and information density of Diplomacy's game state. Combined with the high variance of matches, these factors made Diplomacy prohibitive for study. In this work, we used data-driven iteration to optimize a textual game state representation such that a 24B model can reliably complete matches without any fine tuning. We develop tooling to facilitate hypothesis testing and statistical analysis, and we present case studies on persuasion, aggressive playstyles, and performance across a range of models. We conduct a variety of experiments across many popular LLMs, finding the larger models perform the best, but the smaller models still play adequately. We also introduce Critical State Analysis: an experimental protocol for rapidly iterating and analyzing key moments in a game at depth. Our harness democratizes the evaluation of strategic reasoning in LLMs by eliminating the need for fine-tuning, and it provides insights into how these capabilities emerge naturally from widely used LLMs. Our code is available in the supplement and will be open sourced. |
| title | Democratizing Diplomacy: A Harness for Evaluating Any Large Language Model on Full-Press Diplomacy |
| topic | Artificial Intelligence Computation and Language Computers and Society Machine Learning |
| url | https://arxiv.org/abs/2508.07485 |