Using Large Language Models to Generate JUnit Tests: An Empirical Study
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912004036362240 |
|---|---|
| author | Siddiq, Mohammed Latif Santos, Joanna C. S. Tanvir, Ridwanul Hasan Ulfat, Noshin Rifat, Fahmid Al Lopes, Vinicius Carvalho |
| author_facet | Siddiq, Mohammed Latif Santos, Joanna C. S. Tanvir, Ridwanul Hasan Ulfat, Noshin Rifat, Fahmid Al Lopes, Vinicius Carvalho |
| contents | A code generation model generates code by taking a prompt from a code comment, existing code, or a combination of both. Although code generation models (e.g., GitHub Copilot) are increasingly being adopted in practice, it is unclear whether they can successfully be used for unit test generation without fine-tuning for a strongly typed language like Java. To fill this gap, we investigated how well three models (Codex, GPT-3.5-Turbo, and StarCoder) can generate unit tests. We used two benchmarks (HumanEval and Evosuite SF110) to investigate the effect of context generation on the unit test generation process. We evaluated the models based on compilation rates, test correctness, test coverage, and test smells. We found that the Codex model achieved above 80% coverage for the HumanEval dataset, but no model had more than 2% coverage for the EvoSuite SF110 benchmark. The generated tests also suffered from test smells, such as Duplicated Asserts and Empty Tests. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_00418 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Using Large Language Models to Generate JUnit Tests: An Empirical Study Siddiq, Mohammed Latif Santos, Joanna C. S. Tanvir, Ridwanul Hasan Ulfat, Noshin Rifat, Fahmid Al Lopes, Vinicius Carvalho Software Engineering Machine Learning A code generation model generates code by taking a prompt from a code comment, existing code, or a combination of both. Although code generation models (e.g., GitHub Copilot) are increasingly being adopted in practice, it is unclear whether they can successfully be used for unit test generation without fine-tuning for a strongly typed language like Java. To fill this gap, we investigated how well three models (Codex, GPT-3.5-Turbo, and StarCoder) can generate unit tests. We used two benchmarks (HumanEval and Evosuite SF110) to investigate the effect of context generation on the unit test generation process. We evaluated the models based on compilation rates, test correctness, test coverage, and test smells. We found that the Codex model achieved above 80% coverage for the HumanEval dataset, but no model had more than 2% coverage for the EvoSuite SF110 benchmark. The generated tests also suffered from test smells, such as Duplicated Asserts and Empty Tests. |
| title | Using Large Language Models to Generate JUnit Tests: An Empirical Study |
| topic | Software Engineering Machine Learning |
| url | https://arxiv.org/abs/2305.00418 |