Cover Image

Uncovering Systematic Failures of LLMs in Verifying Code Against Natural Language Specifications

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Jin, Haolin, Chen, Huaming
Format:	Preprint
Published:	2025
Subjects:	Software Engineering Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Are LLMs Reliable Code Reviewers? Systematic Overcorrection in Requirement Conformance Judgement
by: Jin, Haolin, et al.
Published: (2026)

Towards Advancing Code Generation with Large Language Models: A Research Roadmap
by: Jin, Haolin, et al.
Published: (2025)

RGD: Multi-LLM Based Agent Debugger via Refinement and Generation Guidance
by: Jin, Haolin, et al.
Published: (2024)

LLMs are All You Need? Improving Fuzz Testing for MOJO with Large Language Models
by: Huang, Linghan, et al.
Published: (2025)

From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future
by: Jin, Haolin, et al.
Published: (2024)

From Natural Language to Verified Code: Toward AI Assisted Problem-to-Code Generation with Dafny-Based Formal Verification
by: Erfan, Md, et al.
Published: (2026)

An Execution-Verified Multi-Language Benchmark for Code Semantic Reasoning
by: Li, Yikun, et al.
Published: (2026)

On the Challenges of Fuzzing Techniques via Large Language Models
by: Huang, Linghan, et al.
Published: (2024)

What You See Is Not Always What You Get: Evaluating GPT's Comprehension of Source Code
by: Wen, Jiawen, et al.
Published: (2024)

Bridging Generation and Training: A Systematic Review of Quality Issues in LLMs for Code
by: He, Kaifeng, et al.
Published: (2026)

Revisiting the Role of Natural Language Code Comments in Code Translation
by: Gupta, Monika, et al.
Published: (2026)

SIEVE: Towards Verifiable Certification for Code-datasets
by: Mbodji, Fatou Ndiaye, et al.
Published: (2025)

WybeCoder: Verified Imperative Code Generation
by: Gloeckle, Fabian, et al.
Published: (2026)

Uncovering Code Insights: Leveraging GitHub Artifacts for Deeper Code Understanding
by: Nevo, Ziv, et al.
Published: (2025)

CodeMMR: Bridging Natural Language, Code, and Image for Unified Retrieval
by: Geng, Jiahui, et al.
Published: (2026)

Uncovering LLM-Generated Code: A Zero-Shot Synthetic Code Detector via Code Rewriting
by: Ye, Tong, et al.
Published: (2024)

The Specification Gap: Coordination Failure Under Partial Knowledge in Code Agents
by: Sartori, Camilo Chacón
Published: (2026)

Aletheia: What Makes RLVR For Code Verifiers Tick?
by: Venkatkrishna, Vatsal, et al.
Published: (2026)

CodeCrash: Exposing LLM Fragility to Misleading Natural Language in Code Reasoning
by: Lam, Man Ho, et al.
Published: (2025)

Are Large Language Models Robust in Understanding Code Against Semantics-Preserving Mutations?
by: Orvalho, Pedro, et al.
Published: (2025)

MergeRepair: An Exploratory Study on Merging Task-Specific Adapters in Code LLMs for Automated Program Repair
by: Dehghan, Meghdad, et al.
Published: (2024)

VeriAct: Beyond Verifiability -- Agentic Synthesis of Correct and Complete Formal Specifications
by: Misu, Md Rakib Hossain, et al.
Published: (2026)

Edit, But Verify: An Empirical Audit of Instructed Code-Editing Benchmarks
by: Ebrahimi, Amir M., et al.
Published: (2026)

Uncovering Intention through LLM-Driven Code Snippet Description Generation
by: Nugroho, Yusuf Sulistyo, et al.
Published: (2025)

WebVIA: A Web-based Vision-Language Agentic Framework for Interactive and Verifiable UI-to-Code Generation
by: Xu, Mingde, et al.
Published: (2025)

Verifying LLM-Generated Code in the Context of Software Verification with Ada/SPARK
by: Cramer, Marcos, et al.
Published: (2025)

Specification and Detection of LLM Code Smells
by: Mahmoudi, Brahim, et al.
Published: (2025)

Inferring Code Correctness from Specification
by: Florian, Tambon, et al.
Published: (2026)

CLAP: Learning Transferable Binary Code Representations with Natural Language Supervision
by: Wang, Hao, et al.
Published: (2024)

Natural Language based Specification and Verification
by: Li, Zhaorui, et al.
Published: (2026)

CodeGRAG: Bridging the Gap between Natural Language and Programming Language via Graphical Retrieval Augmented Generation
by: Du, Kounianhua, et al.
Published: (2024)

CVeDRL: An Efficient Code Verifier via Difficulty-aware Reinforcement Learning
by: Shi, Ji, et al.
Published: (2026)

Turbulence: Systematically and Automatically Testing Instruction-Tuned Large Language Models for Code
by: Honarvar, Shahin, et al.
Published: (2023)

ML Code Smells: From Specification to Detection
by: Mahmoudi, Brahim, et al.
Published: (2025)

Hallucinations in Code Change to Natural Language Generation: Prevalence and Evaluation of Detection Metrics
by: Liu, Chunhua, et al.
Published: (2025)

When the Specification Emerges: Benchmarking Faithfulness Loss in Long-Horizon Coding Agents
by: Yan, Lu, et al.
Published: (2026)

Benchmarking and Improving Monitors for Out-Of-Distribution Alignment Failure in LLMs
by: Feng, Dylan, et al.
Published: (2026)

AutoICE: Automatically Synthesizing Verifiable C Code via LLM-driven Evolution
by: Luo, Weilin, et al.
Published: (2025)

CodeS: Natural Language to Code Repository via Multi-Layer Sketch
by: Zan, Daoguang, et al.
Published: (2024)

Enhancing Cross-Language Code Translation via Task-Specific Embedding Alignment in Retrieval-Augmented Generation
by: Bhattarai, Manish, et al.
Published: (2024)