LLM-based Vulnerability Discovery through the Lens of Code Metrics
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911172153835520 |
|---|---|
| author | Weissberg, Felix Pirch, Lukas Imgrund, Erik Möller, Jonas Eisenhofer, Thorsten Rieck, Konrad |
| author_facet | Weissberg, Felix Pirch, Lukas Imgrund, Erik Möller, Jonas Eisenhofer, Thorsten Rieck, Konrad |
| contents | Large language models (LLMs) excel in many tasks of software engineering, yet progress in leveraging them for vulnerability discovery has stalled in recent years. To understand this phenomenon, we investigate LLMs through the lens of classic code metrics. Surprisingly, we find that a classifier trained solely on these metrics performs on par with state-of-the-art LLMs for vulnerability discovery. A root-cause analysis reveals a strong correlation and a causal effect between LLMs and code metrics: When the value of a metric is changed, LLM predictions tend to shift by a corresponding magnitude. This dependency suggests that LLMs operate at a similarly shallow level as code metrics, limiting their ability to grasp complex patterns and fully realize their potential in vulnerability discovery. Based on these findings, we derive recommendations on how research should more effectively address this challenge. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_19117 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LLM-based Vulnerability Discovery through the Lens of Code Metrics Weissberg, Felix Pirch, Lukas Imgrund, Erik Möller, Jonas Eisenhofer, Thorsten Rieck, Konrad Cryptography and Security Machine Learning Software Engineering Large language models (LLMs) excel in many tasks of software engineering, yet progress in leveraging them for vulnerability discovery has stalled in recent years. To understand this phenomenon, we investigate LLMs through the lens of classic code metrics. Surprisingly, we find that a classifier trained solely on these metrics performs on par with state-of-the-art LLMs for vulnerability discovery. A root-cause analysis reveals a strong correlation and a causal effect between LLMs and code metrics: When the value of a metric is changed, LLM predictions tend to shift by a corresponding magnitude. This dependency suggests that LLMs operate at a similarly shallow level as code metrics, limiting their ability to grasp complex patterns and fully realize their potential in vulnerability discovery. Based on these findings, we derive recommendations on how research should more effectively address this challenge. |
| title | LLM-based Vulnerability Discovery through the Lens of Code Metrics |
| topic | Cryptography and Security Machine Learning Software Engineering |
| url | https://arxiv.org/abs/2509.19117 |