Are Language Models Efficient Reasoners? A Perspective from Logic Programming
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915730830655488 |
|---|---|
| author | Opedal, Andreas Zengaffinen, Yanick Shirakami, Haruki Pasti, Clemente Sachan, Mrinmaya Saparov, Abulhair Cotterell, Ryan Schölkopf, Bernhard |
| author_facet | Opedal, Andreas Zengaffinen, Yanick Shirakami, Haruki Pasti, Clemente Sachan, Mrinmaya Saparov, Abulhair Cotterell, Ryan Schölkopf, Bernhard |
| contents | Modern language models (LMs) exhibit strong deductive reasoning capabilities, yet standard evaluations emphasize correctness while overlooking a key aspect of reasoning: efficiency. In real-world reasoning scenarios, much of the available information is irrelevant, and effective deductive inference requires identifying and ignoring such distractions. We propose a framework for assessing LM reasoning efficiency through the lens of logic programming, introducing a simple method to align proofs written in natural language -- as generated by an LM -- with shortest proofs found by executing the logic program. Efficiency is quantified by measuring how well a model avoids unnecessary inference. Empirically, we construct a dataset of math word problems injected with various number of irrelevant axioms that vary in semantic overlap with the goal theorem. We find that current LMs show marked accuracy declines under such conditions -- even with minimal, domain-consistent distractions -- and the proofs they generate frequently exhibit detours through irrelevant inferences. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_25626 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Are Language Models Efficient Reasoners? A Perspective from Logic Programming Opedal, Andreas Zengaffinen, Yanick Shirakami, Haruki Pasti, Clemente Sachan, Mrinmaya Saparov, Abulhair Cotterell, Ryan Schölkopf, Bernhard Computation and Language Artificial Intelligence Machine Learning Logic in Computer Science Modern language models (LMs) exhibit strong deductive reasoning capabilities, yet standard evaluations emphasize correctness while overlooking a key aspect of reasoning: efficiency. In real-world reasoning scenarios, much of the available information is irrelevant, and effective deductive inference requires identifying and ignoring such distractions. We propose a framework for assessing LM reasoning efficiency through the lens of logic programming, introducing a simple method to align proofs written in natural language -- as generated by an LM -- with shortest proofs found by executing the logic program. Efficiency is quantified by measuring how well a model avoids unnecessary inference. Empirically, we construct a dataset of math word problems injected with various number of irrelevant axioms that vary in semantic overlap with the goal theorem. We find that current LMs show marked accuracy declines under such conditions -- even with minimal, domain-consistent distractions -- and the proofs they generate frequently exhibit detours through irrelevant inferences. |
| title | Are Language Models Efficient Reasoners? A Perspective from Logic Programming |
| topic | Computation and Language Artificial Intelligence Machine Learning Logic in Computer Science |
| url | https://arxiv.org/abs/2510.25626 |