PATCHEVAL: A New Benchmark for Evaluating LLMs on Patching Real-World Vulnerabilities
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911265783283712 |
|---|---|
| author | Wei, Zichao Zeng, Jun Wen, Ming Yu, Zeliang Cheng, Kai Zhu, Yiding Guo, Jingyi Zhou, Shiqi Yin, Le Su, Xiaodong Ma, Zhechao |
| author_facet | Wei, Zichao Zeng, Jun Wen, Ming Yu, Zeliang Cheng, Kai Zhu, Yiding Guo, Jingyi Zhou, Shiqi Yin, Le Su, Xiaodong Ma, Zhechao |
| contents | Software vulnerabilities are increasing at an alarming rate. However, manual patching is both time-consuming and resource-intensive, while existing automated vulnerability repair (AVR) techniques remain limited in effectiveness. Recent advances in large language models (LLMs) have opened a new paradigm for AVR, demonstrating remarkable progress. To examine the capability of LLMs in AVR, several vulnerability benchmarks have been proposed recently. However, they still suffer from key limitations of outdated vulnerabilities, limited language coverage, unreliable patch validation, and insufficient reproducibility. To overcome these challenges, we introduce PATCHEVAL, a multilingual benchmark for Go, JavaScript, and Python, languages for which existing benchmarks remain unexplored. PATCHEVAL curates a dataset of 1,000 vulnerabilities drawn from CVEs reported between 2015 and 2025, covering 65 distinct CWEs. A subset of 230 CVEs is further equipped with runtime sandbox environments, enabling patch verification through both security tests and functionality tests. To provide a systematic comparison of LLM-based vulnerability repair, we evaluate a series of state-of-the-art LLMs and agents, presenting an in-depth analysis that empirically yields key insights to guide future research in AVR. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_11019 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PATCHEVAL: A New Benchmark for Evaluating LLMs on Patching Real-World Vulnerabilities Wei, Zichao Zeng, Jun Wen, Ming Yu, Zeliang Cheng, Kai Zhu, Yiding Guo, Jingyi Zhou, Shiqi Yin, Le Su, Xiaodong Ma, Zhechao Cryptography and Security Software Engineering Software vulnerabilities are increasing at an alarming rate. However, manual patching is both time-consuming and resource-intensive, while existing automated vulnerability repair (AVR) techniques remain limited in effectiveness. Recent advances in large language models (LLMs) have opened a new paradigm for AVR, demonstrating remarkable progress. To examine the capability of LLMs in AVR, several vulnerability benchmarks have been proposed recently. However, they still suffer from key limitations of outdated vulnerabilities, limited language coverage, unreliable patch validation, and insufficient reproducibility. To overcome these challenges, we introduce PATCHEVAL, a multilingual benchmark for Go, JavaScript, and Python, languages for which existing benchmarks remain unexplored. PATCHEVAL curates a dataset of 1,000 vulnerabilities drawn from CVEs reported between 2015 and 2025, covering 65 distinct CWEs. A subset of 230 CVEs is further equipped with runtime sandbox environments, enabling patch verification through both security tests and functionality tests. To provide a systematic comparison of LLM-based vulnerability repair, we evaluate a series of state-of-the-art LLMs and agents, presenting an in-depth analysis that empirically yields key insights to guide future research in AVR. |
| title | PATCHEVAL: A New Benchmark for Evaluating LLMs on Patching Real-World Vulnerabilities |
| topic | Cryptography and Security Software Engineering |
| url | https://arxiv.org/abs/2511.11019 |