How Clued up are LLMs? Evaluating Multi-Step Deductive Reasoning in a Text-Based Game Environment
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914404774182912 |
|---|---|
| author | Ansell, Rebecca Toney-Wails, Autumn |
| author_facet | Ansell, Rebecca Toney-Wails, Autumn |
| contents | Deducing whodunit proves challenging for LLM agents. In this paper, we implement a text-based multi-agent version of the classic board game Clue as a rule-based testbed for evaluating multi-step deductive reasoning, with six agents drawn from GPT-4o-mini and Gemini-2.5-Flash. We further investigate whether fine-tuning on structured logic puzzles transfers to improved in-game reasoning and gameplay. Across 18 simulated games, agents achieve only four correct wins, indicating difficulty in maintaining consistent deductive reasoning over the course of a full game. Additionally, we find that fine-tuning does not reliably improve performance and, in some cases, appears to increase reasoning volume without improving reasoning precision. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_17169 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | How Clued up are LLMs? Evaluating Multi-Step Deductive Reasoning in a Text-Based Game Environment Ansell, Rebecca Toney-Wails, Autumn Artificial Intelligence Computation and Language I.2.8; I.2.7 Deducing whodunit proves challenging for LLM agents. In this paper, we implement a text-based multi-agent version of the classic board game Clue as a rule-based testbed for evaluating multi-step deductive reasoning, with six agents drawn from GPT-4o-mini and Gemini-2.5-Flash. We further investigate whether fine-tuning on structured logic puzzles transfers to improved in-game reasoning and gameplay. Across 18 simulated games, agents achieve only four correct wins, indicating difficulty in maintaining consistent deductive reasoning over the course of a full game. Additionally, we find that fine-tuning does not reliably improve performance and, in some cases, appears to increase reasoning volume without improving reasoning precision. |
| title | How Clued up are LLMs? Evaluating Multi-Step Deductive Reasoning in a Text-Based Game Environment |
| topic | Artificial Intelligence Computation and Language I.2.8; I.2.7 |
| url | https://arxiv.org/abs/2603.17169 |