Evaluating Large Language Models in Vulnerability Detection Under Variable Context Windows
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913673726918656 |
|---|---|
| author | Lin, Jie Mohaisen, David |
| author_facet | Lin, Jie Mohaisen, David |
| contents | This study examines the impact of tokenized Java code length on the accuracy and explicitness of ten major LLMs in vulnerability detection. Using chi-square tests and known ground truth, we found inconsistencies across models: some, like GPT-4, Mistral, and Mixtral, showed robustness, while others exhibited a significant link between tokenized length and performance. We recommend future LLM development focus on minimizing the influence of input length for better vulnerability detection. Additionally, preprocessing techniques that reduce token count while preserving code structure could enhance LLM accuracy and explicitness in these tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_00064 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Evaluating Large Language Models in Vulnerability Detection Under Variable Context Windows Lin, Jie Mohaisen, David Cryptography and Security Machine Learning This study examines the impact of tokenized Java code length on the accuracy and explicitness of ten major LLMs in vulnerability detection. Using chi-square tests and known ground truth, we found inconsistencies across models: some, like GPT-4, Mistral, and Mixtral, showed robustness, while others exhibited a significant link between tokenized length and performance. We recommend future LLM development focus on minimizing the influence of input length for better vulnerability detection. Additionally, preprocessing techniques that reduce token count while preserving code structure could enhance LLM accuracy and explicitness in these tasks. |
| title | Evaluating Large Language Models in Vulnerability Detection Under Variable Context Windows |
| topic | Cryptography and Security Machine Learning |
| url | https://arxiv.org/abs/2502.00064 |