Cover Image

PATCHEVAL: A New Benchmark for Evaluating LLMs on Patching Real-World Vulnerabilities

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Wei, Zichao, Zeng, Jun, Wen, Ming, Yu, Zeliang, Cheng, Kai, Zhu, Yiding, Guo, Jingyi, Zhou, Shiqi, Yin, Le, Su, Xiaodong, Ma, Zhechao
Format:	Preprint
Published:	2025
Subjects:	Cryptography and Security Software Engineering
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Evaluating LLMs for One-Shot Patching of Real and Artificial Vulnerabilities
by: Garg, Aayush, et al.
Published: (2025)

APPATCH: Automated Adaptive Prompting Large Language Models for Real-World Software Vulnerability Patching
by: Nong, Yu, et al.
Published: (2024)

Revisiting Vulnerability Patch Identification on Data in the Wild
by: Irsan, Ivana Clairine, et al.
Published: (2026)

PatchSeeker: Mapping NVD Records to their Vulnerability-fixing Commits with LLM Generated Commits and Embeddings
by: Nguyen, Huu Hung, et al.
Published: (2025)

Uncovering Hidden Inclusions of Vulnerable Dependencies in Real-World Java Projects
by: Schott, Stefan, et al.
Published: (2026)

Real-World Usability of Vulnerability Proof-of-Concepts: A Comprehensive Study
by: Dang, Wenjing, et al.
Published: (2025)

ContractTinker: LLM-Empowered Vulnerability Repair for Real-World Smart Contracts
by: Wang, Che, et al.
Published: (2024)

From LLMs to Agents: A Comparative Evaluation of LLMs and LLM-based Agents in Security Patch Detection
by: Han, Junxiao, et al.
Published: (2025)

Towards Understanding and Characterizing Vulnerabilities in Intelligent Connected Vehicles through Real-World Exploits
by: Wang, Yuelin, et al.
Published: (2026)

A Slicing-Based Approach for Detecting and Patching Vulnerable Code Clones
by: Alomari, Hakam, et al.
Published: (2025)

Similar but Patched Code Considered Harmful -- The Impact of Similar but Patched Code on Recurring Vulnerability Detection and How to Remove Them
by: Tan, Zixuan, et al.
Published: (2024)

RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories
by: Wang, Yanlin, et al.
Published: (2026)

Mono: Is Your "Clean" Vulnerability Dataset Really Solvable? Exposing and Trapping Undecidable Patches and Beyond
by: Gao, Zeyu, et al.
Published: (2025)

Match & Mend: Minimally Invasive Local Reassembly for Patching N-day Vulnerabilities in ARM Binaries
by: Jänich, Sebastian, et al.
Published: (2025)

From Trace to Line: LLM Agent for Real-World OSS Vulnerability Localization
by: Xi, Haoran, et al.
Published: (2025)

Vulnerability Patching Across Software Products and Software Components: A Case Study of Red Hat's Product Portfolio
by: Ruohonen, Jukka, et al.
Published: (2025)

Lightweight Vulnerability Detection from Code Metrics and Token Features
by: Chiu, Chun Yin
Published: (2026)

Guiding Symbolic Execution with Static Analysis and LLMs for Vulnerability Discovery
by: Shafiuzzaman, Md, et al.
Published: (2026)

HYDRA: A Hybrid Heuristic-Guided Deep Representation Architecture for Predicting Latent Zero-Day Vulnerabilities in Patched Functions
by: Farhad, Mohammad, et al.
Published: (2025)

Do Fine-Tuned LLMs Understand Vulnerabilities? An Investigation into the Semantic Trap
by: Huang, Feiyang, et al.
Published: (2026)

Game Rewards Vulnerabilities: Software Vulnerability Detection with Zero-Sum Game and Prototype Learning
by: Wen, Xin-Cheng, et al.
Published: (2024)

PatchFuzz: Patch Fuzzing for JavaScript Engines
by: Wang, Junjie, et al.
Published: (2025)

Cross-Ecosystem Vulnerability Analysis for Python Applications
by: Alexopoulos, Georgios, et al.
Published: (2026)

From Transactions to Exploits: Automated PoC Synthesis for Real-World DeFi Attacks
by: Su, Xing, et al.
Published: (2026)

Vulnerability Detection with Interprocedural Context in Multiple Languages: Assessing Effectiveness and Cost of Modern LLMs
by: Lira, Kevin, et al.
Published: (2026)

Fast and Accurate Silent Vulnerability Fix Retrieval
by: Liu, Xueqing, et al.
Published: (2025)

Zero-Shot Vulnerability Detection in Low-Resource Smart Contracts Through Solidity-Only Training
by: Hu, Minghao, et al.
Published: (2026)

Smart Contract Fuzzing Towards Profitable Vulnerabilities
by: Kong, Ziqiao, et al.
Published: (2025)

VulInstruct: Teaching LLMs Root-Cause Reasoning for Vulnerability Detection via Security Specifications
by: Zhu, Hao, et al.
Published: (2025)

Out of Distribution, Out of Luck: How Well Can LLMs Trained on Vulnerability Datasets Detect Top 25 CWE Weaknesses?
by: Li, Yikun, et al.
Published: (2025)

BackportBench: A Multilingual Benchmark for Automated Backporting of Patches
by: Zhong, Zhiqing, et al.
Published: (2025)

Are Latent Vulnerabilities Hidden Gems for Software Vulnerability Prediction? An Empirical Study
by: Le, Triet H. M., et al.
Published: (2024)

VulKey: Automated Vulnerability Repair Guided by Domain-Specific Repair Patterns
by: Li, Jia, et al.
Published: (2026)

TPSQLi: Test Prioritization for SQL Injection Vulnerability Detection in Web Applications
by: Yang, Guan-Yan, et al.
Published: (2025)

Harnessing the Power of LLMs in Source Code Vulnerability Detection
by: Mahyari, Andrew A
Published: (2024)

Closing the Gap: A User Study on the Real-world Usefulness of AI-powered Vulnerability Detection & Repair in the IDE
by: Steenhoek, Benjamin, et al.
Published: (2024)

Beyond Function-Level Analysis: Context-Aware Reasoning for Inter-Procedural Vulnerability Detection
by: Li, Yikun, et al.
Published: (2026)

StriderSPD: Structure-Guided Joint Representation Learning for Binary Security Patch Detection
by: Li, Qingyuan, et al.
Published: (2026)

LLMs for Cyber Security: New Opportunities
by: Divakaran, Dinil Mon, et al.
Published: (2024)

On the Effectiveness of Function-Level Vulnerability Detectors for Inter-Procedural Vulnerabilities
by: Li, Zhen, et al.
Published: (2024)