Bayesian Social Deduction with Graph-Informed Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911581971939328 |
|---|---|
| author | Rahimirad, Shahab Gergerli, Guven Romero, Lucia Qian, Angela Olson, Matthew Lyle Stepputtis, Simon Campbell, Joseph |
| author_facet | Rahimirad, Shahab Gergerli, Guven Romero, Lucia Qian, Angela Olson, Matthew Lyle Stepputtis, Simon Campbell, Joseph |
| contents | Social reasoning - inferring unobservable beliefs and intentions from partial observations of other agents - remains a challenging task for large language models (LLMs). We evaluate the limits of current reasoning language models in the social deduction game Avalon and find that while the largest models demonstrate strong performance, they require extensive test-time inference and degrade sharply when distilled to smaller, real-time-capable variants. To address this, we introduce a hybrid reasoning framework that externalizes belief inference to a structured probabilistic model, while using an LLM for language understanding and interaction. Our approach achieves competitive performance with much larger models in Agent-Agent play and, notably, is the first language agent to defeat human players in a controlled study - achieving a 67% win rate and receiving higher qualitative ratings than both reasoning baselines and human teammates. We release code, models, and a dataset to support future work on social reasoning in LLM agents, which can be found at https://camp-lab-purdue.github.io/bayesian-social-deduction/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_17788 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Bayesian Social Deduction with Graph-Informed Language Models Rahimirad, Shahab Gergerli, Guven Romero, Lucia Qian, Angela Olson, Matthew Lyle Stepputtis, Simon Campbell, Joseph Artificial Intelligence Computation and Language Machine Learning Multiagent Systems I.2.1; I.2.7 Social reasoning - inferring unobservable beliefs and intentions from partial observations of other agents - remains a challenging task for large language models (LLMs). We evaluate the limits of current reasoning language models in the social deduction game Avalon and find that while the largest models demonstrate strong performance, they require extensive test-time inference and degrade sharply when distilled to smaller, real-time-capable variants. To address this, we introduce a hybrid reasoning framework that externalizes belief inference to a structured probabilistic model, while using an LLM for language understanding and interaction. Our approach achieves competitive performance with much larger models in Agent-Agent play and, notably, is the first language agent to defeat human players in a controlled study - achieving a 67% win rate and receiving higher qualitative ratings than both reasoning baselines and human teammates. We release code, models, and a dataset to support future work on social reasoning in LLM agents, which can be found at https://camp-lab-purdue.github.io/bayesian-social-deduction/ |
| title | Bayesian Social Deduction with Graph-Informed Language Models |
| topic | Artificial Intelligence Computation and Language Machine Learning Multiagent Systems I.2.1; I.2.7 |
| url | https://arxiv.org/abs/2506.17788 |