TurnaboutLLM: A Deductive Reasoning Benchmark from Detective Games
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915505506353152 |
|---|---|
| author | Yuan, Yuan He, Muyu Shahid, Muhammad Adil Huang, Jiani Li, Ziyang Zhang, Li |
| author_facet | Yuan, Yuan He, Muyu Shahid, Muhammad Adil Huang, Jiani Li, Ziyang Zhang, Li |
| contents | This paper introduces TurnaboutLLM, a novel framework and dataset for evaluating the deductive reasoning abilities of Large Language Models (LLMs) by leveraging the interactive gameplay of detective games Ace Attorney and Danganronpa. The framework tasks LLMs with identifying contradictions between testimonies and evidences within long narrative contexts, a challenging task due to the large answer space and diverse reasoning types presented by its questions. We evaluate twelve state-of-the-art LLMs on the dataset, hinting at limitations of popular strategies for enhancing deductive reasoning such as extensive thinking and Chain-of-Thought prompting. The results also suggest varying effects of context size, the number of reasoning step and answer space size on model performance. Overall, TurnaboutLLM presents a substantial challenge for LLMs' deductive reasoning abilities in complex, narrative-rich environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_15712 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | TurnaboutLLM: A Deductive Reasoning Benchmark from Detective Games Yuan, Yuan He, Muyu Shahid, Muhammad Adil Huang, Jiani Li, Ziyang Zhang, Li Computation and Language This paper introduces TurnaboutLLM, a novel framework and dataset for evaluating the deductive reasoning abilities of Large Language Models (LLMs) by leveraging the interactive gameplay of detective games Ace Attorney and Danganronpa. The framework tasks LLMs with identifying contradictions between testimonies and evidences within long narrative contexts, a challenging task due to the large answer space and diverse reasoning types presented by its questions. We evaluate twelve state-of-the-art LLMs on the dataset, hinting at limitations of popular strategies for enhancing deductive reasoning such as extensive thinking and Chain-of-Thought prompting. The results also suggest varying effects of context size, the number of reasoning step and answer space size on model performance. Overall, TurnaboutLLM presents a substantial challenge for LLMs' deductive reasoning abilities in complex, narrative-rich environments. |
| title | TurnaboutLLM: A Deductive Reasoning Benchmark from Detective Games |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.15712 |