The Prompt Alchemist: Automated LLM-Tailored Prompt Optimization for Test Case Generation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929655710220288 |
|---|---|
| author | Gao, Shuzheng Wang, Chaozheng Gao, Cuiyun Jiao, Xiaoqian Chong, Chun Yong Gao, Shan Lyu, Michael |
| author_facet | Gao, Shuzheng Wang, Chaozheng Gao, Cuiyun Jiao, Xiaoqian Chong, Chun Yong Gao, Shan Lyu, Michael |
| contents | Test cases are essential for validating the reliability and quality of software applications. Recent studies have demonstrated the capability of Large Language Models (LLMs) to generate useful test cases for given source code. However, the existing work primarily relies on human-written plain prompts, which often leads to suboptimal results since the performance of LLMs can be highly influenced by the prompts. Moreover, these approaches use the same prompt for all LLMs, overlooking the fact that different LLMs might be best suited to different prompts. Given the wide variety of possible prompt formulations, automatically discovering the optimal prompt for each LLM presents a significant challenge. Although there are methods on automated prompt optimization in the natural language processing field, they are hard to produce effective prompts for the test case generation task. First, the methods iteratively optimize prompts by simply combining and mutating existing ones without proper guidance, resulting in prompts that lack diversity and tend to repeat the same errors in the generated test cases. Second, the prompts are generally lack of domain contextual knowledge, limiting LLMs' performance in the task. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_01329 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | The Prompt Alchemist: Automated LLM-Tailored Prompt Optimization for Test Case Generation Gao, Shuzheng Wang, Chaozheng Gao, Cuiyun Jiao, Xiaoqian Chong, Chun Yong Gao, Shan Lyu, Michael Software Engineering Artificial Intelligence Computation and Language Test cases are essential for validating the reliability and quality of software applications. Recent studies have demonstrated the capability of Large Language Models (LLMs) to generate useful test cases for given source code. However, the existing work primarily relies on human-written plain prompts, which often leads to suboptimal results since the performance of LLMs can be highly influenced by the prompts. Moreover, these approaches use the same prompt for all LLMs, overlooking the fact that different LLMs might be best suited to different prompts. Given the wide variety of possible prompt formulations, automatically discovering the optimal prompt for each LLM presents a significant challenge. Although there are methods on automated prompt optimization in the natural language processing field, they are hard to produce effective prompts for the test case generation task. First, the methods iteratively optimize prompts by simply combining and mutating existing ones without proper guidance, resulting in prompts that lack diversity and tend to repeat the same errors in the generated test cases. Second, the prompts are generally lack of domain contextual knowledge, limiting LLMs' performance in the task. |
| title | The Prompt Alchemist: Automated LLM-Tailored Prompt Optimization for Test Case Generation |
| topic | Software Engineering Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2501.01329 |