Reporting LLM Prompting in Automated Software Engineering: A Guideline Based on Current Practices and Expectations
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917184337346560 |
|---|---|
| author | Korn, Alexander Zaruchas, Lea Arora, Chetan Metzger, Andreas Smolka, Sven Wang, Fanyu Vogelsang, Andreas |
| author_facet | Korn, Alexander Zaruchas, Lea Arora, Chetan Metzger, Andreas Smolka, Sven Wang, Fanyu Vogelsang, Andreas |
| contents | Large Language Models, particularly decoder-only generative models such as GPT, are increasingly used to automate Software Engineering tasks. These models are primarily guided through natural language prompts, making prompt engineering a critical factor in system performance and behavior. Despite their growing role in SE research, prompt-related decisions are rarely documented in a systematic or transparent manner, hindering reproducibility and comparability across studies. To address this gap, we conducted a two-phase empirical study. First, we analyzed nearly 300 papers published at the top-3 SE conferences since 2022 to assess how prompt design, testing, and optimization are currently reported. Second, we surveyed 105 program committee members from these conferences to capture their expectations for prompt reporting in LLM-driven research. Based on the findings, we derived a structured guideline that distinguishes essential, desirable, and exceptional reporting elements. Our results reveal significant misalignment between current practices and reviewer expectations, particularly regarding version disclosure, prompt justification, and threats to validity. We present our guideline as a step toward improving transparency, reproducibility, and methodological rigor in LLM-based SE research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_01954 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Reporting LLM Prompting in Automated Software Engineering: A Guideline Based on Current Practices and Expectations Korn, Alexander Zaruchas, Lea Arora, Chetan Metzger, Andreas Smolka, Sven Wang, Fanyu Vogelsang, Andreas Software Engineering Large Language Models, particularly decoder-only generative models such as GPT, are increasingly used to automate Software Engineering tasks. These models are primarily guided through natural language prompts, making prompt engineering a critical factor in system performance and behavior. Despite their growing role in SE research, prompt-related decisions are rarely documented in a systematic or transparent manner, hindering reproducibility and comparability across studies. To address this gap, we conducted a two-phase empirical study. First, we analyzed nearly 300 papers published at the top-3 SE conferences since 2022 to assess how prompt design, testing, and optimization are currently reported. Second, we surveyed 105 program committee members from these conferences to capture their expectations for prompt reporting in LLM-driven research. Based on the findings, we derived a structured guideline that distinguishes essential, desirable, and exceptional reporting elements. Our results reveal significant misalignment between current practices and reviewer expectations, particularly regarding version disclosure, prompt justification, and threats to validity. We present our guideline as a step toward improving transparency, reproducibility, and methodological rigor in LLM-based SE research. |
| title | Reporting LLM Prompting in Automated Software Engineering: A Guideline Based on Current Practices and Expectations |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2601.01954 |