LineBreaker: Finding Token-Inconsistency Bugs with Large Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909838105116672 |
|---|---|
| author | Chen, Hongbo Zhang, Yifan Han, Xing Mao, Tianhao Rong, Huanyao Zhang, Yuheng Wang, XiaoFeng Xing, Luyi Chen, Xun Zhang, Hang |
| author_facet | Chen, Hongbo Zhang, Yifan Han, Xing Mao, Tianhao Rong, Huanyao Zhang, Yuheng Wang, XiaoFeng Xing, Luyi Chen, Xun Zhang, Hang |
| contents | Token-inconsistency bugs (TIBs) involve the misuse of syntactically valid yet incorrect code tokens, such as misused variables and erroneous function invocations, which can often lead to software bugs. Unlike simple syntactic bugs, TIBs occur at the semantic level and are subtle - sometimes they remain undetected for years. Traditional detection methods, such as static analysis and dynamic testing, often struggle with TIBs due to their versatile and context-dependent nature. However, advancements in large language models (LLMs) like GPT-4 present new opportunities for automating TIB detection by leveraging these models' semantic understanding capabilities.
This paper reports the first systematic measurement of LLMs' capabilities in detecting TIBs, revealing that while GPT-4 shows promise, it exhibits limitations in precision and scalability. Specifically, its detection capability is undermined by the model's tendency to focus on the code snippets that do not contain TIBs; its scalability concern arises from GPT-4's high cost and the massive amount of code requiring inspection. To address these challenges, we introduce \name, a novel and cascaded TIB detection system. \name leverages smaller, code-specific, and highly efficient language models to filter out large numbers of code snippets unlikely to contain TIBs, thereby significantly enhancing the system's performance in terms of precision, recall, and scalability. We evaluated \name on 154 Python and C GitHub repositories, each with over 1,000 stars, uncovering 123 new flaws, 45\% of which could be exploited to disrupt program functionalities. Out of our 69 submitted fixes, 41 have already been confirmed or merged. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_01668 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LineBreaker: Finding Token-Inconsistency Bugs with Large Language Models Chen, Hongbo Zhang, Yifan Han, Xing Mao, Tianhao Rong, Huanyao Zhang, Yuheng Wang, XiaoFeng Xing, Luyi Chen, Xun Zhang, Hang Cryptography and Security Software Engineering Token-inconsistency bugs (TIBs) involve the misuse of syntactically valid yet incorrect code tokens, such as misused variables and erroneous function invocations, which can often lead to software bugs. Unlike simple syntactic bugs, TIBs occur at the semantic level and are subtle - sometimes they remain undetected for years. Traditional detection methods, such as static analysis and dynamic testing, often struggle with TIBs due to their versatile and context-dependent nature. However, advancements in large language models (LLMs) like GPT-4 present new opportunities for automating TIB detection by leveraging these models' semantic understanding capabilities. This paper reports the first systematic measurement of LLMs' capabilities in detecting TIBs, revealing that while GPT-4 shows promise, it exhibits limitations in precision and scalability. Specifically, its detection capability is undermined by the model's tendency to focus on the code snippets that do not contain TIBs; its scalability concern arises from GPT-4's high cost and the massive amount of code requiring inspection. To address these challenges, we introduce \name, a novel and cascaded TIB detection system. \name leverages smaller, code-specific, and highly efficient language models to filter out large numbers of code snippets unlikely to contain TIBs, thereby significantly enhancing the system's performance in terms of precision, recall, and scalability. We evaluated \name on 154 Python and C GitHub repositories, each with over 1,000 stars, uncovering 123 new flaws, 45\% of which could be exploited to disrupt program functionalities. Out of our 69 submitted fixes, 41 have already been confirmed or merged. |
| title | LineBreaker: Finding Token-Inconsistency Bugs with Large Language Models |
| topic | Cryptography and Security Software Engineering |
| url | https://arxiv.org/abs/2405.01668 |