Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.29512 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914612826341376 |
|---|---|
| author | Wang, Kevin Thöni, Anna Kempinski, Benjamin Cheng, Bobby Yao, Jianzhu Finch, Benjamin Guertler, Leon Nadkarni, Viraj Jiang, Yihan Korshuk, Aliaksei Buyantuev, Alexander Makarov, Ilya Wu, Siyuan Cheng, Yu-Chi Ju, Yan-Ru Wu, Ti-Rong Chu, I-Hsuan Yang, Yu-Yu Wu, I-Chen Huang, Yitian Cao, Qinlu Sun, Yiheng Dai, Yuhong Yao, Hongkun Fu, Jingxuan Zhang, Jiwei Liao, Hao Ebeling, Mossimo Arun, Govind Bathini, Sadhvik Arya, Mihir S Anish, Avinash Ranjan, Aditya Phatnani, Kirtana Sunil KS, Paval Mehta, Vrushali S, Aravind Arora, Nikhil Upadhyay, Tanya Bandagale, Amol Lu, Yuan Hsiao, ChunEn Lin, YuTing Chung, Arvin Thomas, Jerry John Laurière, Mathieu Choshen, Leshem Bachrach, Yoram Viswanath, Pramod Polukarov, Maria Tan, Cheston Kachman, Tal Wang, Atlas |
| author_facet | Wang, Kevin Thöni, Anna Kempinski, Benjamin Cheng, Bobby Yao, Jianzhu Finch, Benjamin Guertler, Leon Nadkarni, Viraj Jiang, Yihan Korshuk, Aliaksei Buyantuev, Alexander Makarov, Ilya Wu, Siyuan Cheng, Yu-Chi Ju, Yan-Ru Wu, Ti-Rong Chu, I-Hsuan Yang, Yu-Yu Wu, I-Chen Huang, Yitian Cao, Qinlu Sun, Yiheng Dai, Yuhong Yao, Hongkun Fu, Jingxuan Zhang, Jiwei Liao, Hao Ebeling, Mossimo Arun, Govind Bathini, Sadhvik Arya, Mihir S Anish, Avinash Ranjan, Aditya Phatnani, Kirtana Sunil KS, Paval Mehta, Vrushali S, Aravind Arora, Nikhil Upadhyay, Tanya Bandagale, Amol Lu, Yuan Hsiao, ChunEn Lin, YuTing Chung, Arvin Thomas, Jerry John Laurière, Mathieu Choshen, Leshem Bachrach, Yoram Viswanath, Pramod Polukarov, Maria Tan, Cheston Kachman, Tal Wang, Atlas |
| contents | Large language models (LLMs) are increasingly deployed as interactive agents, yet their capacity for social and strategic reasoning over extended interaction remains poorly understood. Existing evaluations rely on static vignettes or single-game benchmarks that cannot capture the sustained, multi-faceted reasoning that real-world multi-agent settings demand. We introduce Mindgames, a multi-game arena and evaluation platform for LLM agents that operationalizes complementary reasoning demands relevant to ``theory of mind'': belief attribution under hidden information, opponent modeling through repeated strategic interaction, cooperative inference under knowledge asymmetries, and sustained deception in social deduction. Built on TextArena, Mindgames provides a unified interaction interface, TrueSkill-based rating, and full trajectory logging across four game environments. We instantiate Mindgames through a 2025 competition cycle hosted at a major AI conference, which assessed 944 submitted agents from 76 teams across four games: Colonel Blotto, Iterated Prisoner's Dilemma, Codenames, and Secret Mafia. Our analysis surfaces both agent-level and evaluation-level limitations: brittle rule adherence remains a major bottleneck, top-performing systems repeatedly rely on explicit structural scaffolding, and leaderboard validity differs sharply across environments. In particular, failure-heavy environments can reward robustness to opponent errors as much as strategic ability, with Secret Mafia exhibiting a pronounced error-survival confound in this cycle. We release a dataset of 29,571 multi-agent games with turn-level observations, actions, and rewards, together with MG-Ref, a deterministic offline tournament protocol that scores new agents against a frozen reference pool of top-ranked, low-error Stage~II submissions under the same error-attribution lens used in this analysis. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_29512 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MINDGAMES: A Live Arena for Evaluating Social and Strategic Reasoning in Multi-Agent LLMs Wang, Kevin Thöni, Anna Kempinski, Benjamin Cheng, Bobby Yao, Jianzhu Finch, Benjamin Guertler, Leon Nadkarni, Viraj Jiang, Yihan Korshuk, Aliaksei Buyantuev, Alexander Makarov, Ilya Wu, Siyuan Cheng, Yu-Chi Ju, Yan-Ru Wu, Ti-Rong Chu, I-Hsuan Yang, Yu-Yu Wu, I-Chen Huang, Yitian Cao, Qinlu Sun, Yiheng Dai, Yuhong Yao, Hongkun Fu, Jingxuan Zhang, Jiwei Liao, Hao Ebeling, Mossimo Arun, Govind Bathini, Sadhvik Arya, Mihir S Anish, Avinash Ranjan, Aditya Phatnani, Kirtana Sunil KS, Paval Mehta, Vrushali S, Aravind Arora, Nikhil Upadhyay, Tanya Bandagale, Amol Lu, Yuan Hsiao, ChunEn Lin, YuTing Chung, Arvin Thomas, Jerry John Laurière, Mathieu Choshen, Leshem Bachrach, Yoram Viswanath, Pramod Polukarov, Maria Tan, Cheston Kachman, Tal Wang, Atlas Artificial Intelligence Large language models (LLMs) are increasingly deployed as interactive agents, yet their capacity for social and strategic reasoning over extended interaction remains poorly understood. Existing evaluations rely on static vignettes or single-game benchmarks that cannot capture the sustained, multi-faceted reasoning that real-world multi-agent settings demand. We introduce Mindgames, a multi-game arena and evaluation platform for LLM agents that operationalizes complementary reasoning demands relevant to ``theory of mind'': belief attribution under hidden information, opponent modeling through repeated strategic interaction, cooperative inference under knowledge asymmetries, and sustained deception in social deduction. Built on TextArena, Mindgames provides a unified interaction interface, TrueSkill-based rating, and full trajectory logging across four game environments. We instantiate Mindgames through a 2025 competition cycle hosted at a major AI conference, which assessed 944 submitted agents from 76 teams across four games: Colonel Blotto, Iterated Prisoner's Dilemma, Codenames, and Secret Mafia. Our analysis surfaces both agent-level and evaluation-level limitations: brittle rule adherence remains a major bottleneck, top-performing systems repeatedly rely on explicit structural scaffolding, and leaderboard validity differs sharply across environments. In particular, failure-heavy environments can reward robustness to opponent errors as much as strategic ability, with Secret Mafia exhibiting a pronounced error-survival confound in this cycle. We release a dataset of 29,571 multi-agent games with turn-level observations, actions, and rewards, together with MG-Ref, a deterministic offline tournament protocol that scores new agents against a frozen reference pool of top-ranked, low-error Stage~II submissions under the same error-attribution lens used in this analysis. |
| title | MINDGAMES: A Live Arena for Evaluating Social and Strategic Reasoning in Multi-Agent LLMs |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2605.29512 |