Revisiting the Performance of Deep Learning-Based Vulnerability Detection on Realistic Datasets
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909240975687680 |
|---|---|
| author | Chakraborty, Partha Arumugam, Krishna Kanth Alfadel, Mahmoud Nagappan, Meiyappan McIntosh, Shane |
| author_facet | Chakraborty, Partha Arumugam, Krishna Kanth Alfadel, Mahmoud Nagappan, Meiyappan McIntosh, Shane |
| contents | The impact of software vulnerabilities on everyday software systems is significant. Despite deep learning models being proposed for vulnerability detection, their reliability is questionable. Prior evaluations show high recall/F1 scores of up to 99%, but these models underperform in practical scenarios, particularly when assessed on entire codebases rather than just the fixing commit. This paper introduces Real-Vul, a comprehensive dataset representing real-world scenarios for evaluating vulnerability detection models. Evaluating DeepWukong, LineVul, ReVeal, and IVDetect shows a significant drop in performance, with precision decreasing by up to 95 percentage points and F1 scores by up to 91 points. Furthermore, Model performance fluctuates based on vulnerability characteristics, with better F1 scores for information leaks or code injection than for path resolution or predictable return values. The results highlight a significant performance gap that needs addressing before deploying deep learning-based vulnerability detection in practical settings. Overfitting is identified as a key issue, and an augmentation technique is proposed, potentially improving performance by up to 30%. Contributions include a dataset creation approach for better model evaluation, Real-Vul dataset, and empirical evidence of deep learning models struggling in real-world settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_03093 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Revisiting the Performance of Deep Learning-Based Vulnerability Detection on Realistic Datasets Chakraborty, Partha Arumugam, Krishna Kanth Alfadel, Mahmoud Nagappan, Meiyappan McIntosh, Shane Software Engineering Artificial Intelligence Cryptography and Security Machine Learning D.2; I.2 The impact of software vulnerabilities on everyday software systems is significant. Despite deep learning models being proposed for vulnerability detection, their reliability is questionable. Prior evaluations show high recall/F1 scores of up to 99%, but these models underperform in practical scenarios, particularly when assessed on entire codebases rather than just the fixing commit. This paper introduces Real-Vul, a comprehensive dataset representing real-world scenarios for evaluating vulnerability detection models. Evaluating DeepWukong, LineVul, ReVeal, and IVDetect shows a significant drop in performance, with precision decreasing by up to 95 percentage points and F1 scores by up to 91 points. Furthermore, Model performance fluctuates based on vulnerability characteristics, with better F1 scores for information leaks or code injection than for path resolution or predictable return values. The results highlight a significant performance gap that needs addressing before deploying deep learning-based vulnerability detection in practical settings. Overfitting is identified as a key issue, and an augmentation technique is proposed, potentially improving performance by up to 30%. Contributions include a dataset creation approach for better model evaluation, Real-Vul dataset, and empirical evidence of deep learning models struggling in real-world settings. |
| title | Revisiting the Performance of Deep Learning-Based Vulnerability Detection on Realistic Datasets |
| topic | Software Engineering Artificial Intelligence Cryptography and Security Machine Learning D.2; I.2 |
| url | https://arxiv.org/abs/2407.03093 |