RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912438177234944 |
|---|---|
| author | Xu, Xinnuo Lawrence, Rachel Dubey, Kshitij Pandey, Atharva Ueno, Risa Falck, Fabian Nori, Aditya V. Sharma, Rahul Sharma, Amit Gonzalez, Javier |
| author_facet | Xu, Xinnuo Lawrence, Rachel Dubey, Kshitij Pandey, Atharva Ueno, Risa Falck, Fabian Nori, Aditya V. Sharma, Rahul Sharma, Amit Gonzalez, Javier |
| contents | Recent Large Language Models (LLMs) have reported high accuracy on reasoning benchmarks. However, it is still unclear whether the observed results arise from true reasoning or from statistical recall of the training set. Inspired by the ladder of causation (Pearl, 2009) and its three levels (associations, interventions and counterfactuals), this paper introduces RE-IMAGINE, a framework to characterize a hierarchy of reasoning ability in LLMs, alongside an automated pipeline to generate problem variations at different levels of the hierarchy. By altering problems in an intermediate symbolic representation, RE-IMAGINE generates arbitrarily many problems that are not solvable using memorization alone. Moreover, the framework is general and can work across reasoning domains, including math, code, and logic. We demonstrate our framework on four widely-used benchmarks to evaluate several families of LLMs, and observe reductions in performance when the models are queried with problem variations. These assessments indicate a degree of reliance on statistical recall for past performance, and open the door to further research targeting skills across the reasoning hierarchy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_15455 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation Xu, Xinnuo Lawrence, Rachel Dubey, Kshitij Pandey, Atharva Ueno, Risa Falck, Fabian Nori, Aditya V. Sharma, Rahul Sharma, Amit Gonzalez, Javier Computation and Language Artificial Intelligence Recent Large Language Models (LLMs) have reported high accuracy on reasoning benchmarks. However, it is still unclear whether the observed results arise from true reasoning or from statistical recall of the training set. Inspired by the ladder of causation (Pearl, 2009) and its three levels (associations, interventions and counterfactuals), this paper introduces RE-IMAGINE, a framework to characterize a hierarchy of reasoning ability in LLMs, alongside an automated pipeline to generate problem variations at different levels of the hierarchy. By altering problems in an intermediate symbolic representation, RE-IMAGINE generates arbitrarily many problems that are not solvable using memorization alone. Moreover, the framework is general and can work across reasoning domains, including math, code, and logic. We demonstrate our framework on four widely-used benchmarks to evaluate several families of LLMs, and observe reductions in performance when the models are queried with problem variations. These assessments indicate a degree of reliance on statistical recall for past performance, and open the door to further research targeting skills across the reasoning hierarchy. |
| title | RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2506.15455 |