On the Evaluation of Large Language Models in Unit Test Generation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912044895174656 |
|---|---|
| author | Yang, Lin Yang, Chen Gao, Shutao Wang, Weijing Wang, Bo Zhu, Qihao Chu, Xiao Zhou, Jianyi Liang, Guangtai Wang, Qianxiang Chen, Junjie |
| author_facet | Yang, Lin Yang, Chen Gao, Shutao Wang, Weijing Wang, Bo Zhu, Qihao Chu, Xiao Zhou, Jianyi Liang, Guangtai Wang, Qianxiang Chen, Junjie |
| contents | Unit testing is an essential activity in software development for verifying the correctness of software components. However, manually writing unit tests is challenging and time-consuming. The emergence of Large Language Models (LLMs) offers a new direction for automating unit test generation. Existing research primarily focuses on closed-source LLMs (e.g., ChatGPT and CodeX) with fixed prompting strategies, leaving the capabilities of advanced open-source LLMs with various prompting settings unexplored. Particularly, open-source LLMs offer advantages in data privacy protection and have demonstrated superior performance in some tasks. Moreover, effective prompting is crucial for maximizing LLMs' capabilities. In this paper, we conduct the first empirical study to fill this gap, based on 17 Java projects, five widely-used open-source LLMs with different structures and parameter sizes, and comprehensive evaluation metrics. Our findings highlight the significant influence of various prompt factors, show the performance of open-source LLMs compared to the commercial GPT-4 and the traditional Evosuite, and identify limitations in LLM-based unit test generation. We then derive a series of implications from our study to guide future research and practical use of LLM-based unit test generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_18181 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | On the Evaluation of Large Language Models in Unit Test Generation Yang, Lin Yang, Chen Gao, Shutao Wang, Weijing Wang, Bo Zhu, Qihao Chu, Xiao Zhou, Jianyi Liang, Guangtai Wang, Qianxiang Chen, Junjie Software Engineering Unit testing is an essential activity in software development for verifying the correctness of software components. However, manually writing unit tests is challenging and time-consuming. The emergence of Large Language Models (LLMs) offers a new direction for automating unit test generation. Existing research primarily focuses on closed-source LLMs (e.g., ChatGPT and CodeX) with fixed prompting strategies, leaving the capabilities of advanced open-source LLMs with various prompting settings unexplored. Particularly, open-source LLMs offer advantages in data privacy protection and have demonstrated superior performance in some tasks. Moreover, effective prompting is crucial for maximizing LLMs' capabilities. In this paper, we conduct the first empirical study to fill this gap, based on 17 Java projects, five widely-used open-source LLMs with different structures and parameter sizes, and comprehensive evaluation metrics. Our findings highlight the significant influence of various prompt factors, show the performance of open-source LLMs compared to the commercial GPT-4 and the traditional Evosuite, and identify limitations in LLM-based unit test generation. We then derive a series of implications from our study to guide future research and practical use of LLM-based unit test generation. |
| title | On the Evaluation of Large Language Models in Unit Test Generation |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2406.18181 |