Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.15503 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910247023542272 |
|---|---|
| author | Baltes, Sebastian Angermeir, Florian Arora, Chetan Barón, Marvin Muñoz Chen, Chunyang Böhme, Lukas Calefato, Fabio Ernst, Neil Falessi, Davide Fitzgerald, Brian Fucci, Davide He, Junda Treude, Christoph Kalinowski, Marcos Lambiase, Stefano Russo, Daniel Lungu, Mircea Montes, Cristina Martinez Prechelt, Lutz Ralph, Paul van Tonder, Rijnard Wagner, Stefan |
| author_facet | Baltes, Sebastian Angermeir, Florian Arora, Chetan Barón, Marvin Muñoz Chen, Chunyang Böhme, Lukas Calefato, Fabio Ernst, Neil Falessi, Davide Fitzgerald, Brian Fucci, Davide He, Junda Treude, Christoph Kalinowski, Marcos Lambiase, Stefano Russo, Daniel Lungu, Mircea Montes, Cristina Martinez Prechelt, Lutz Ralph, Paul van Tonder, Rijnard Wagner, Stefan |
| contents | Large Language Models (LLMs) are widely used in software engineering (SE) research and practice, yet their non-determinism, opaque training data, and rapidly evolving models threaten the reproducibility and replicability of empirical studies. We address this challenge through a collaborative effort of 22 researchers, presenting a taxonomy of seven study types that organizes how LLMs are used in SE research, together with eight guidelines for designing and reporting such studies. Each guideline distinguishes requirements (must) from recommended practices (should) and is contextualized by the study types it applies to. Our guidelines recommend that researchers: (1) declare LLM usage and role; (2) report model versions, configurations, and customizations; (3) document the tool architecture beyond the model; (4) disclose prompts, their development, and interaction logs; (5) validate LLM outputs with humans; (6) include an open LLM as a baseline; (7) use suitable baselines, benchmarks, and metrics; and (8) articulate limitations and mitigations. We complement the guidelines with an applicability matrix mapping guidelines to study types and a reporting checklist for authors and reviewers. We maintain the study types and guidelines online as a living resource for the community to use and shape (llm-guidelines$.$org). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_15503 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Guidelines for Empirical Studies in Software Engineering involving Large Language Models Baltes, Sebastian Angermeir, Florian Arora, Chetan Barón, Marvin Muñoz Chen, Chunyang Böhme, Lukas Calefato, Fabio Ernst, Neil Falessi, Davide Fitzgerald, Brian Fucci, Davide He, Junda Treude, Christoph Kalinowski, Marcos Lambiase, Stefano Russo, Daniel Lungu, Mircea Montes, Cristina Martinez Prechelt, Lutz Ralph, Paul van Tonder, Rijnard Wagner, Stefan Software Engineering Large Language Models (LLMs) are widely used in software engineering (SE) research and practice, yet their non-determinism, opaque training data, and rapidly evolving models threaten the reproducibility and replicability of empirical studies. We address this challenge through a collaborative effort of 22 researchers, presenting a taxonomy of seven study types that organizes how LLMs are used in SE research, together with eight guidelines for designing and reporting such studies. Each guideline distinguishes requirements (must) from recommended practices (should) and is contextualized by the study types it applies to. Our guidelines recommend that researchers: (1) declare LLM usage and role; (2) report model versions, configurations, and customizations; (3) document the tool architecture beyond the model; (4) disclose prompts, their development, and interaction logs; (5) validate LLM outputs with humans; (6) include an open LLM as a baseline; (7) use suitable baselines, benchmarks, and metrics; and (8) articulate limitations and mitigations. We complement the guidelines with an applicability matrix mapping guidelines to study types and a reporting checklist for authors and reviewers. We maintain the study types and guidelines online as a living resource for the community to use and shape (llm-guidelines$.$org). |
| title | Guidelines for Empirical Studies in Software Engineering involving Large Language Models |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2508.15503 |