Cover Image

Socrates or Smartypants: Testing Logic Reasoning Capabilities of Large Language Models with Logic Programming-based Test Oracles

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Xu, Zihao, Ding, Junchen, Lou, Yiling, Zhang, Kun, Gong, Dong, Li, Yuekang
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

"Pull or Not to Pull?'': Investigating Moral Biases in Leading Large Language Models Across Ethical Dilemmas
by: Ding, Junchen, et al.
Published: (2025)

Disentangling Logic: The Role of Context in Large Language Model Reasoning Capabilities
by: Hua, Wenyue, et al.
Published: (2024)

Lost in the Logic: An Evaluation of Large Language Models' Reasoning Capabilities on LSAT Logic Games
by: Malik, Saumya
Published: (2024)

TombRaider: Entering the Vault of History to Jailbreak Large Language Models
by: Ding, Junchen, et al.
Published: (2025)

Logic-of-Thought: Injecting Logic into Contexts for Full Reasoning in Large Language Models
by: Liu, Tongxuan, et al.
Published: (2024)

Logical Reasoning in Large Language Models: A Survey
by: Liu, Hanmeng, et al.
Published: (2025)

On Memorization of Large Language Models in Logical Reasoning
by: Xie, Chulin, et al.
Published: (2024)

GLoRE: Evaluating Logical Reasoning of Large Language Models
by: liu, Hanmeng, et al.
Published: (2023)

Logical Reasoning with Outcome Reward Models for Test-Time Scaling
by: Thatikonda, Ramya Keerthy, et al.
Published: (2025)

From Program Logics to Language Logics
by: Cimini, Matteo
Published: (2024)

Reason from Fallacy: Enhancing Large Language Models' Logical Reasoning through Logical Fallacy Understanding
by: Li, Yanda, et al.
Published: (2024)

CLR-Fact: Evaluating the Complex Logical Reasoning Capability of Large Language Models over Factual Knowledge
by: Zheng, Tianshi, et al.
Published: (2024)

Logic Contrastive Reasoning with Lightweight Large Language Model for Math Word Problems
by: Kai, Ding, et al.
Published: (2024)

Knowledge Vector of Logical Reasoning in Large Language Models
by: Wang, Zixuan, et al.
Published: (2026)

Language Models and Logic Programs for Trustworthy Tax Reasoning
by: Jurayj, William, et al.
Published: (2025)

A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoning
by: Hong, Ruixin, et al.
Published: (2023)

ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models
by: Tang, Liyan, et al.
Published: (2025)

LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models
by: Parmar, Mihir, et al.
Published: (2024)

Causal Language Modeling Can Elicit Search and Reasoning Capabilities on Logic Puzzles
by: Shah, Kulin, et al.
Published: (2024)

LogiNumSynth: Synthesizing Joint Logical-Numerical Reasoning Problems for Language Models
by: Liu, Yiwei, et al.
Published: (2025)

Chain of Logic: Rule-Based Reasoning with Large Language Models
by: Servantez, Sergio, et al.
Published: (2024)

Enhancing Logical Reasoning in Large Language Models through Graph-based Synthetic Data
by: Zhou, Jiaming, et al.
Published: (2024)

LogicPro: Improving Complex Logical Reasoning via Program-Guided Learning
by: Jiang, Jin, et al.
Published: (2024)

Large Language Models Meet Symbolic Provers for Logical Reasoning Evaluation
by: Qi, Chengwen, et al.
Published: (2025)

Reasoning-as-Logic-Units: Scaling Test-Time Reasoning in Large Language Models Through Logic Unit Alignment
by: Li, Cheryl, et al.
Published: (2025)

Assessing Logical Reasoning Capabilities of Encoder-Only Transformer Models
by: Pirozelli, Paulo, et al.
Published: (2023)

LogicAsker: Evaluating and Improving the Logical Reasoning Ability of Large Language Models
by: Wan, Yuxuan, et al.
Published: (2024)

Enhancing Zero-Shot Chain-of-Thought Reasoning in Large Language Models through Logic
by: Zhao, Xufeng, et al.
Published: (2023)

LogicGame: Benchmarking Rule-Based Reasoning Abilities of Large Language Models
by: Gui, Jiayi, et al.
Published: (2024)

Do Large Language Models Excel in Complex Logical Reasoning with Formal Language?
by: Jiang, Jin, et al.
Published: (2025)

DivLogicEval: A Framework for Benchmarking Logical Reasoning Evaluation in Large Language Models
by: Chung, Tsz Ting, et al.
Published: (2025)

LogicTree: Structured Proof Exploration for Coherent and Rigorous Logical Reasoning with Large Language Models
by: He, Kang, et al.
Published: (2025)

Image-Based Geolocation Using Large Vision-Language Models
by: Liu, Yi, et al.
Published: (2024)

Cerisier: A Program Logic for Attestation in a Capability Machine
by: Rousseau, June, et al.
Published: (2026)

Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMs
by: Wang, Siyuan, et al.
Published: (2024)

CLOMO: Counterfactual Logical Modification with Large Language Models
by: Huang, Yinya, et al.
Published: (2023)

Last Layer Logits to Logic: Empowering LLMs with Logic-Consistent Structured Knowledge Reasoning
by: Li, Songze, et al.
Published: (2025)

Case Study: Testing Model Capabilities in Some Reasoning Tasks
by: Zhang, Min, et al.
Published: (2024)

A Large Language Model Based Method for Complex Logical Reasoning over Knowledge Graphs
by: Zhang, Ziyan, et al.
Published: (2025)

Adaptable Logical Control for Large Language Models
by: Zhang, Honghua, et al.
Published: (2024)