Cover Image

Can Language Models Falsify? Evaluating Algorithmic Reasoning with Counterexample Creation

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Sinha, Shiven, Goel, Shashwat, Kumaraguru, Ponnurangam, Geiping, Jonas, Bethge, Matthias, Prabhu, Ameya
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Software Engineering
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Wu's Method can Boost Symbolic AI to Rival Silver Medalists and AlphaGeometry to Outperform Gold Medalists at IMO Geometry
by: Sinha, Shiven, et al.
Published: (2024)

PostTrainBench: Can LLM Agents Automate LLM Post-Training?
by: Rank, Ben, et al.
Published: (2026)

Great Models Think Alike and this Undermines AI Oversight
by: Goel, Shashwat, et al.
Published: (2025)

Answer Matching Outperforms Multiple Choice for Language Model Evaluation
by: Chandak, Nikhil, et al.
Published: (2025)

Scaling Open-Ended Reasoning to Predict the Future
by: Chandak, Nikhil, et al.
Published: (2025)

Corrective Machine Unlearning
by: Goel, Shashwat, et al.
Published: (2024)

Random Representations Outperform Online Continually Learned Representations
by: Prabhu, Ameya, et al.
Published: (2024)

Small Models, Big Tasks: An Exploratory Empirical Study on Small Language Models for Function Calling
by: Kavathekar, Ishan, et al.
Published: (2025)

Can Code Language Models Learn Clarification-Seeking Behaviors?
by: Wu, Jie JW, et al.
Published: (2025)

Pitfalls in Evaluating Language Model Forecasters
by: Paleka, Daniel, et al.
Published: (2025)

Evaluating the Generalization Capabilities of Large Language Models on Code Reasoning
by: Yang, Rem, et al.
Published: (2025)

Can OpenSource beat ChatGPT? -- A Comparative Study of Large Language Models for Text-to-Code Generation
by: Mayer, Luis, et al.
Published: (2024)

Towards Continuous Assurance Case Creation for ADS with the Evidential Tool Bus
by: Sorokin, Lev, et al.
Published: (2024)

EsoLang-Bench: Evaluating Genuine Reasoning in Large Language Models via Esoteric Programming Languages
by: Sharma, Aman, et al.
Published: (2026)

Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks
by: Li, Yuangang, et al.
Published: (2026)

AlgoTune: Can Language Models Speed Up General-Purpose Numerical Programs?
by: Press, Ori, et al.
Published: (2025)

InfiBench: Evaluating the Question-Answering Capabilities of Code Large Language Models
by: Li, Linyi, et al.
Published: (2024)

AssertionBench: A Benchmark to Evaluate Large-Language Models for Assertion Generation
by: Pulavarthi, Vaishnavi, et al.
Published: (2024)

TDD-Bench Verified: Can LLMs Generate Tests for Issues Before They Get Resolved?
by: Ahmed, Toufique, et al.
Published: (2024)

Can LLMs Find Bugs in Code? An Evaluation from Beginner Errors to Security Vulnerabilities in Python and C++
by: Mhatre, Akshay, et al.
Published: (2025)

Strategic Dishonesty Can Undermine AI Safety Evaluations of Frontier LLMs
by: Panfilov, Alexander, et al.
Published: (2025)

Improving the Reproducibility of Deep Learning Software: An Initial Investigation through a Case Study Analysis
by: Ravi, Nikita, et al.
Published: (2025)

David vs. Goliath: Can Small Models Win Big with Agentic AI in Hardware Design?
by: Shankar, Shashwat, et al.
Published: (2025)

What if I ask in \textit{alia lingua}? Measuring Functional Similarity Across Languages
by: Mishra, Debangan, et al.
Published: (2025)

Large Language Models for Code Generation: A Comprehensive Survey of Challenges, Techniques, Evaluation, and Applications
by: Huynh, Nam, et al.
Published: (2025)

IDE-Bench: Evaluating Large Language Models as IDE Agents on Real-World Software Engineering Tasks
by: Mateega, Spencer, et al.
Published: (2026)

Can Hessian-Based Insights Support Fault Diagnosis in Attention-based Models?
by: Jahan, Sigma, et al.
Published: (2025)

NESSiE: The Necessary Safety Benchmark -- Identifying Errors that should not Exist
by: Bertram, Johannes, et al.
Published: (2026)

Can LLMs Reason Like Automated Theorem Provers for Rust Verification? VCoT-Bench: Evaluating via Verification Chain of Thought
by: Xie, Zichen, et al.
Published: (2026)

HackerRank-ASTRA: Evaluating Correctness & Consistency of Large Language Models on cross-domain multi-file project problems
by: Xing, Jun, et al.
Published: (2025)

iServe: An Intent-based Serving System for LLMs
by: Liakopoulos, Dimitrios, et al.
Published: (2025)

The Counterfeit Conundrum: Can Code Language Models Grasp the Nuances of Their Incorrect Generations?
by: Gu, Alex, et al.
Published: (2024)

FutureSim: Replaying World Events to Evaluate Adaptive Agents
by: Goel, Shashwat, et al.
Published: (2026)

Design, Implementation and Evaluation of a Novel Programming Language Topic Classification Workflow
by: Zhang, Michael, et al.
Published: (2025)

Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructions
by: Cassano, Federico, et al.
Published: (2023)

Intrinsic Credit Assignment for Long Horizon Interaction
by: Auzina, Ilze Amanda, et al.
Published: (2026)

Calibration and Correctness of Language Models for Code
by: Spiess, Claudio, et al.
Published: (2024)

Otter: Generating Tests from Issues to Validate SWE Patches
by: Ahmed, Toufique, et al.
Published: (2025)

Evaluating Language Models for Efficient Code Generation
by: Liu, Jiawei, et al.
Published: (2024)

Evaluating the Formal Reasoning Capabilities of Large Language Models through Chomsky Hierarchy
by: Dong, Yihong, et al.
Published: (2026)