Cover Image

CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specification Generation

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Chen, Zaoyu, Dai, Jianbo, Zhu, Boyu, Wang, Jingdong, Wang, Huiming, Xu, Xin, Yuan, Haoyang, Guo, Zhijiang, Wu, Xiao-Ming
Format:	Preprint
Published:	2026
Subjects:	Software Engineering Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

SWE-Bench+: Enhanced Coding Benchmark for LLMs
by: Aleithan, Reem, et al.
Published: (2024)

ResearchEnvBench: Benchmarking Agents on Environment Synthesis for Research Code Execution
by: Wang, Yubang, et al.
Published: (2026)

DiffSpec: Differential Testing with LLMs using Natural Language Specifications and Code Artifacts
by: Rao, Nikitha, et al.
Published: (2024)

SpecEval: Evaluating Code Comprehension in Large Language Models via Program Specifications
by: Ma, Lezhi, et al.
Published: (2024)

QuanBench: Benchmarking Quantum Code Generation with Large Language Models
by: Guo, Xiaoyu, et al.
Published: (2025)

SelfPiCo: Self-Guided Partial Code Execution with LLMs
by: Xue, Zhipeng, et al.
Published: (2024)

CodeBenchGen: Creating Scalable Execution-based Code Generation Benchmarks
by: Xie, Yiqing, et al.
Published: (2024)

OSS-Bench: Benchmark Generator for Coding LLMs
by: Jiang, Yuancheng, et al.
Published: (2025)

Integrating Symbolic Execution with LLMs for Automated Generation of Program Specifications
by: Yang, Fanpeng, et al.
Published: (2025)

EffiLearner: Enhancing Efficiency of Generated Code via Self-Optimization
by: Huang, Dong, et al.
Published: (2024)

EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluations
by: Li, Jia, et al.
Published: (2024)

Compiling Code LLMs into Lightweight Executables
by: Shi, Jieke, et al.
Published: (2026)

Defects4Log: Benchmarking LLMs for Logging Code Defect Detection and Reasoning
by: Wang, Xin, et al.
Published: (2025)

SpecRover: Code Intent Extraction via LLMs
by: Ruan, Haifeng, et al.
Published: (2024)

CodeFuse-CR-Bench: A Comprehensiveness-aware Benchmark for End-to-End Code Review Evaluation in Python Projects
by: Guo, Hanyang, et al.
Published: (2025)

SpecTra: Enhancing the Code Translation Ability of Language Models by Generating Multi-Modal Specifications
by: Nitin, Vikram, et al.
Published: (2024)

DyPyBench: A Benchmark of Executable Python Software
by: Bouzenia, Islem, et al.
Published: (2024)

DesignBench: A Comprehensive Benchmark for MLLM-based Front-end Code Generation
by: Xiao, Jingyu, et al.
Published: (2025)

DSCodeBench: A Realistic Benchmark for Data Science Code Generation
by: Ouyang, Shuyin, et al.
Published: (2025)

RedCode: Risky Code Execution and Generation Benchmark for Code Agents
by: Guo, Chengquan, et al.
Published: (2024)

EffiCoder: Enhancing Code Generation in Large Language Models through Efficiency-Aware Fine-tuning
by: Huang, Dong, et al.
Published: (2024)

VerifyThisBench: Generating Code, Specifications, and Proofs All at Once
by: Deng, Xun, et al.
Published: (2025)

FeedbackEval: A Benchmark for Evaluating Large Language Models in Feedback-Driven Code Repair Tasks
by: Dai, Dekun, et al.
Published: (2025)

STEPWISE-CODEX-Bench: Evaluating Complex Multi-Function Comprehension and Fine-Grained Execution Reasoning
by: Yan, Kaiwen, et al.
Published: (2025)

JMigBench: A Benchmark for Evaluating LLMs on Source Code Migration (Java 8 to Java 11)
by: Amin, Nishil, et al.
Published: (2026)

SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents
by: Zhao, Bingchen, et al.
Published: (2026)

FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation
by: Li, Wei, et al.
Published: (2025)

Towards Automated Smart Contract Generation: Evaluation, Benchmarking, and Retrieval-Augmented Repair
by: Chen, Zaoyu, et al.
Published: (2025)

SpecSyn: LLM-based Synthesis and Refinement of Formal Specifications for Real-world Program Verification
by: Ma, Lezhi, et al.
Published: (2026)

CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation
by: Yan, Weixiang, et al.
Published: (2023)

SafeToolBench: Pioneering a Prospective Benchmark to Evaluating Tool Utilization Safety in LLMs
by: Xia, Hongfei, et al.
Published: (2025)

RubberDuckBench: A Benchmark for AI Coding Assistants
by: Mohammed, Ferida, et al.
Published: (2026)

CodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verification
by: Tian, Yuchen, et al.
Published: (2024)

CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation
by: Wang, Sizhe, et al.
Published: (2025)

Learning Adaptive Parallel Execution for Efficient Code Localization
by: Xu, Ke, et al.
Published: (2026)

Code Digital Twin: Empowering LLMs with Tacit Knowledge for Complex Software Development
by: Peng, Xin, et al.
Published: (2025)

Logging Like Humans for LLMs: Rethinking Logging via Execution and Runtime Feedback
by: Wang, Xin, et al.
Published: (2026)

FeatureBench: Benchmarking Agentic Coding for Complex Feature Development
by: Zhou, Qixing, et al.
Published: (2026)

SpecGen: Automated Generation of Formal Program Specifications via Large Language Models
by: Ma, Lezhi, et al.
Published: (2024)

Accelerating Patch Validation for Program Repair with Interception-Based Execution Scheduling
by: Xiao, Yuan-An, et al.
Published: (2023)