CleanVul: Automatic Function-Level Vulnerability Detection in Code Commits Using LLM Heuristics
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909781565898752 |
|---|---|
| author | Li, Yikun Zhang, Ting Widyasari, Ratnadira Tun, Yan Naing Nguyen, Huu Hung Bui, Tan Irsan, Ivana Clairine Cheng, Yiran Lan, Xiang Ang, Han Wei Liauw, Frank Weyssow, Martin Kang, Hong Jin Ouh, Eng Lieh Shar, Lwin Khin Lo, David |
| author_facet | Li, Yikun Zhang, Ting Widyasari, Ratnadira Tun, Yan Naing Nguyen, Huu Hung Bui, Tan Irsan, Ivana Clairine Cheng, Yiran Lan, Xiang Ang, Han Wei Liauw, Frank Weyssow, Martin Kang, Hong Jin Ouh, Eng Lieh Shar, Lwin Khin Lo, David |
| contents | Accurate identification of software vulnerabilities is crucial for system integrity. Vulnerability datasets, often derived from the National Vulnerability Database (NVD) or directly from GitHub, are essential for training machine learning models to detect these security flaws. However, these datasets frequently suffer from significant noise, typically 40% to 75%, due primarily to the automatic and indiscriminate labeling of all changes in vulnerability-fixing commits (VFCs) as vulnerability-related. This misclassification occurs because not all changes in a commit aimed at fixing vulnerabilities pertain to security threats; many are routine updates like bug fixes or test improvements.
This paper introduces the first methodology that uses the Large Language Model (LLM) with a heuristic enhancement to automatically identify vulnerability-fixing changes from VFCs, achieving an F1-score of 0.82. VulSifter was applied to a large-scale study, where we conducted a crawl of 127,063 repositories on GitHub, resulting in the acquisition of 5,352,105 commits. VulSifter involves utilizing an LLM to comprehend code semantics and contextual information, while applying heuristics to filter out unrelated changes. We then developed CleanVul, a high-quality dataset comprising 8,198 functions using our LLM heuristic enhancement approach, demonstrating Correctness (90.6%) comparable to established datasets such as SVEN and PrimeVul.
To evaluate the CleanVul dataset, we conducted experiments focusing on fine-tuning various LLMs on CleanVul and other high-quality datasets. Evaluation results reveal that LLMs fine-tuned on CleanVul not only exhibit enhanced accuracy but also superior generalization capabilities compared to those trained on uncleaned datasets. Specifically, models trained on CleanVul and tested on PrimeVul achieve accuracy higher than those trained and tested exclusively on PrimeVul. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_17274 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CleanVul: Automatic Function-Level Vulnerability Detection in Code Commits Using LLM Heuristics Li, Yikun Zhang, Ting Widyasari, Ratnadira Tun, Yan Naing Nguyen, Huu Hung Bui, Tan Irsan, Ivana Clairine Cheng, Yiran Lan, Xiang Ang, Han Wei Liauw, Frank Weyssow, Martin Kang, Hong Jin Ouh, Eng Lieh Shar, Lwin Khin Lo, David Software Engineering Cryptography and Security Accurate identification of software vulnerabilities is crucial for system integrity. Vulnerability datasets, often derived from the National Vulnerability Database (NVD) or directly from GitHub, are essential for training machine learning models to detect these security flaws. However, these datasets frequently suffer from significant noise, typically 40% to 75%, due primarily to the automatic and indiscriminate labeling of all changes in vulnerability-fixing commits (VFCs) as vulnerability-related. This misclassification occurs because not all changes in a commit aimed at fixing vulnerabilities pertain to security threats; many are routine updates like bug fixes or test improvements. This paper introduces the first methodology that uses the Large Language Model (LLM) with a heuristic enhancement to automatically identify vulnerability-fixing changes from VFCs, achieving an F1-score of 0.82. VulSifter was applied to a large-scale study, where we conducted a crawl of 127,063 repositories on GitHub, resulting in the acquisition of 5,352,105 commits. VulSifter involves utilizing an LLM to comprehend code semantics and contextual information, while applying heuristics to filter out unrelated changes. We then developed CleanVul, a high-quality dataset comprising 8,198 functions using our LLM heuristic enhancement approach, demonstrating Correctness (90.6%) comparable to established datasets such as SVEN and PrimeVul. To evaluate the CleanVul dataset, we conducted experiments focusing on fine-tuning various LLMs on CleanVul and other high-quality datasets. Evaluation results reveal that LLMs fine-tuned on CleanVul not only exhibit enhanced accuracy but also superior generalization capabilities compared to those trained on uncleaned datasets. Specifically, models trained on CleanVul and tested on PrimeVul achieve accuracy higher than those trained and tested exclusively on PrimeVul. |
| title | CleanVul: Automatic Function-Level Vulnerability Detection in Code Commits Using LLM Heuristics |
| topic | Software Engineering Cryptography and Security |
| url | https://arxiv.org/abs/2411.17274 |