Cover Image

Evaluation of the Programming Skills of Large Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Heitz, Luc Bryan, Chamas, Joun, Scherb, Christopher
Format:	Preprint
Published:	2024
Subjects:	Software Engineering Computation and Language Cryptography and Security
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Large Language Models Cannot Reliably Detect Vulnerabilities in JavaScript: The First Systematic Benchmark and Evaluation
by: Fei, Qingyuan, et al.
Published: (2025)

PoLLMgraph: Unraveling Hallucinations in Large Language Models via State Transition Dynamics
by: Zhu, Derui, et al.
Published: (2024)

Evaluating and Mitigating Linguistic Discrimination in Large Language Models
by: Dong, Guoliang, et al.
Published: (2024)

CyMed: A Framework for Testing Cybersecurity of Connected Medical Devices
by: Scherb, Christopher, et al.
Published: (2023)

AutoTestForge: A Multidimensional Automated Testing Framework for Natural Language Processing Models
by: Xing, Hengrui, et al.
Published: (2025)

Efficient Detection of Toxic Prompts in Large Language Models
by: Liu, Yi, et al.
Published: (2024)

Large Language Models-Aided Program Debloating
by: Lin, Bo, et al.
Published: (2025)

Understanding the Effectiveness of Large Language Models in Detecting Security Vulnerabilities
by: Khare, Avishree, et al.
Published: (2023)

TOSSS: a CVE-based Software Security Benchmark for Large Language Models
by: Damie, Marc, et al.
Published: (2026)

Divide and Conquer based Symbolic Vulnerability Detection
by: Scherb, Christopher, et al.
Published: (2024)

UniTSyn: A Large-Scale Dataset Capable of Enhancing the Prowess of Large Language Models for Program Testing
by: He, Yifeng, et al.
Published: (2024)

Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale
by: Liu, Yi, et al.
Published: (2026)

Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models
by: Dong, Yihong, et al.
Published: (2024)

ContraFix: Agentic Vulnerability Repair via Differential Runtime Evidence and Skill Reuse
by: Liu, Simiao, et al.
Published: (2026)

Evaluating Large Language Models for Line-Level Vulnerability Localization
by: Zhang, Jian, et al.
Published: (2024)

Evaluating Large Language Models in detecting Secrets in Android Apps
by: Alecci, Marco, et al.
Published: (2025)

ProSec: Fortifying Code LLMs with Proactive Security Alignment
by: Xu, Xiangzhe, et al.
Published: (2024)

AC4: Algebraic Computation Checker for Circuit Constraints in ZKPs
by: Yang, Qizhe, et al.
Published: (2024)

Automated Trustworthiness Oracle Generation for Machine Learning Text Classifiers
by: Tung, Lam Nguyen, et al.
Published: (2024)

Jailbreak Distillation: Renewable Safety Benchmarking
by: Zhang, Jingyu, et al.
Published: (2025)

BackportBench: A Multilingual Benchmark for Automated Backporting of Patches
by: Zhong, Zhiqing, et al.
Published: (2025)

Argus: Reorchestrating Static Analysis via a Multi-Agent Ensemble for Full-Chain Security Vulnerability Detection
by: Liang, Zi, et al.
Published: (2026)

Minimal Prompt Perturbations Lead to Code Vulnerabilities: Prompt Fragility and Hidden-State Signals in Coding LLMs
by: Sternfeld, Alexander, et al.
Published: (2026)

Vulnerability Detection in Popular Programming Languages with Language Models
by: Atiiq, Syafiq Al, et al.
Published: (2024)

Static Deadlock Detection for Rust Programs
by: Zhang, Yu, et al.
Published: (2024)

Training Language Model Agents to Find Vulnerabilities with CTF-Dojo
by: Zhuo, Terry Yue, et al.
Published: (2025)

Accelerating Automatic Program Repair with Dual Retrieval-Augmented Fine-Tuning and Patch Generation on Large Language Models
by: Guo, Hanyang, et al.
Published: (2025)

Leveraging Large Language Models for Preliminary Security Risk Analysis: A Mission-Critical Case Study
by: Esposito, Matteo, et al.
Published: (2024)

LUNA: A Model-Based Universal Analysis Framework for Large Language Models
by: Song, Da, et al.
Published: (2023)

VulnRepairEval: An Exploit-Based Evaluation Framework for Assessing Large Language Model Vulnerability Repair Capabilities
by: Wang, Weizhe, et al.
Published: (2025)

LeCov: Multi-level Testing Criteria for Large Language Models
by: Xie, Xuan, et al.
Published: (2024)

Automata-Based Steering of Large Language Models for Diverse Structured Generation
by: Luan, Xiaokun, et al.
Published: (2025)

Divide, Conquer and Verify: Improving Symbolic Execution Performance
by: Scherb, Christopher, et al.
Published: (2023)

A Broad Comparative Evaluation of Software Debloating Tools
by: Brown, Michael D., et al.
Published: (2023)

Large Language Model assisted Hybrid Fuzzing
by: Meng, Ruijie, et al.
Published: (2024)

Large Language Models for Code: Security Hardening and Adversarial Testing
by: He, Jingxuan, et al.
Published: (2023)

Detecting Malicious Intents in Smart Contracts with Pre-trained Programming Language Models
by: Huang, Youwei, et al.
Published: (2025)

The Secrets Must Not Flow: Scaling Security Verification to Large Codebases (extended version)
by: Arquint, Linard, et al.
Published: (2025)

On the effectiveness of Large Language Models for GitHub Workflows
by: Zhang, Xinyu, et al.
Published: (2024)

CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion
by: Ren, Qibing, et al.
Published: (2024)