Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2606.00103 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911734519824384 |
|---|---|
| author | Fan, Mingyuan Han, Weiguang Wang, Daixin Chen, Cen Zhang, Zhiqiang Zhou, Jun |
| author_facet | Fan, Mingyuan Han, Weiguang Wang, Daixin Chen, Cen Zhang, Zhiqiang Zhou, Jun |
| contents | We introduce a multi-turn interactive framework for reasoning evaluation that treats reasoning as active evidence acquisition and belief updating. Wherein, LLMs receive only the task rules, must issue targeted queries to a hidden environment, integrate partial observations over time, and decide when to submit a final answer. Beyond standard success rate and interaction efficiency, we evaluate contextual robustness under controlled contextual perturbations, and metacognitive adaptation through counterfactual revision and necessity judgment. We instantiate the framework as a benchmark of 474 executable games, each evaluated under five fixed configuration search spaces corresponding to five difficulty levels, and evaluate a broad set of frontier LLMs. Results show that the benchmark is highly discriminative, exposing large differences not only in success rate but also in interaction efficiency. Moreover, we empirically show that contextual perturbations cause moderate but consistent declines, whereas counterfactual revision and necessity judgment lead to much larger drops. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2606_00103 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Evaluating Interactive Reasoning in Large Language Models: A Hierarchical Benchmark with Executable Games Fan, Mingyuan Han, Weiguang Wang, Daixin Chen, Cen Zhang, Zhiqiang Zhou, Jun Artificial Intelligence We introduce a multi-turn interactive framework for reasoning evaluation that treats reasoning as active evidence acquisition and belief updating. Wherein, LLMs receive only the task rules, must issue targeted queries to a hidden environment, integrate partial observations over time, and decide when to submit a final answer. Beyond standard success rate and interaction efficiency, we evaluate contextual robustness under controlled contextual perturbations, and metacognitive adaptation through counterfactual revision and necessity judgment. We instantiate the framework as a benchmark of 474 executable games, each evaluated under five fixed configuration search spaces corresponding to five difficulty levels, and evaluate a broad set of frontier LLMs. Results show that the benchmark is highly discriminative, exposing large differences not only in success rate but also in interaction efficiency. Moreover, we empirically show that contextual perturbations cause moderate but consistent declines, whereas counterfactual revision and necessity judgment lead to much larger drops. |
| title | Evaluating Interactive Reasoning in Large Language Models: A Hierarchical Benchmark with Executable Games |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2606.00103 |