LLM-Powered Test Case Generation for Detecting Bugs in Plausible Programs
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916769531166720 |
|---|---|
| author | Liu, Kaibo Chen, Zhenpeng Liu, Yiyang Zhang, Jie M. Harman, Mark Han, Yudong Ma, Yun Dong, Yihong Li, Ge Huang, Gang |
| author_facet | Liu, Kaibo Chen, Zhenpeng Liu, Yiyang Zhang, Jie M. Harman, Mark Han, Yudong Ma, Yun Dong, Yihong Li, Ge Huang, Gang |
| contents | Detecting tricky bugs in plausible programs, those that pass existing test suites yet still contain bugs, remains a significant challenge in software testing. To address this problem, we propose TrickCatcher, an LLM-powered approach to generating test cases for uncovering bugs in plausible programs. TrickCatcher operates in three stages: First, it uses an LLM to generate program variants based on the program under test (PUT) and its specification. Second, it employs an LLM to construct an input generator from the specification for producing test inputs. Finally, these inputs are executed on both the PUT and its program variants to detect inconsistencies in their outputs. We evaluate TrickCatcher on two datasets, TrickyBugs and EvalPlus, which include 366 human-written and 151 AI-generated plausible programs with tricky bugs. TrickCatcher achieves recall, precision, and F1 scores that are 1.80x, 2.65x, and 1.66x those of the state-of-the-art baselines, respectively. Code and data used are available at https://github.com/RinCloud/TrickCatcher. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_10304 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LLM-Powered Test Case Generation for Detecting Bugs in Plausible Programs Liu, Kaibo Chen, Zhenpeng Liu, Yiyang Zhang, Jie M. Harman, Mark Han, Yudong Ma, Yun Dong, Yihong Li, Ge Huang, Gang Software Engineering Machine Learning Detecting tricky bugs in plausible programs, those that pass existing test suites yet still contain bugs, remains a significant challenge in software testing. To address this problem, we propose TrickCatcher, an LLM-powered approach to generating test cases for uncovering bugs in plausible programs. TrickCatcher operates in three stages: First, it uses an LLM to generate program variants based on the program under test (PUT) and its specification. Second, it employs an LLM to construct an input generator from the specification for producing test inputs. Finally, these inputs are executed on both the PUT and its program variants to detect inconsistencies in their outputs. We evaluate TrickCatcher on two datasets, TrickyBugs and EvalPlus, which include 366 human-written and 151 AI-generated plausible programs with tricky bugs. TrickCatcher achieves recall, precision, and F1 scores that are 1.80x, 2.65x, and 1.66x those of the state-of-the-art baselines, respectively. Code and data used are available at https://github.com/RinCloud/TrickCatcher. |
| title | LLM-Powered Test Case Generation for Detecting Bugs in Plausible Programs |
| topic | Software Engineering Machine Learning |
| url | https://arxiv.org/abs/2404.10304 |