Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.07668 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912218451279872 |
|---|---|
| author | Wagner, Stefan Barón, Marvin Muñoz Falessi, Davide Baltes, Sebastian |
| author_facet | Wagner, Stefan Barón, Marvin Muñoz Falessi, Davide Baltes, Sebastian |
| contents | In the short period since the release of ChatGPT, large language models (LLMs) have changed the software engineering research landscape. While there are numerous opportunities to use LLMs for supporting research or software engineering tasks, solid science needs rigorous empirical evaluations. However, so far, there are no specific guidelines for conducting and assessing studies involving LLMs in software engineering research. Our focus is on empirical studies that either use LLMs as part of the research process or studies that evaluate existing or new tools that are based on LLMs. This paper contributes the first set of holistic guidelines for such studies. Our goal is to start a discussion in the software engineering research community to reach a common understanding of our standards for high-quality empirical studies involving LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_07668 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards Evaluation Guidelines for Empirical Studies involving LLMs Wagner, Stefan Barón, Marvin Muñoz Falessi, Davide Baltes, Sebastian Software Engineering In the short period since the release of ChatGPT, large language models (LLMs) have changed the software engineering research landscape. While there are numerous opportunities to use LLMs for supporting research or software engineering tasks, solid science needs rigorous empirical evaluations. However, so far, there are no specific guidelines for conducting and assessing studies involving LLMs in software engineering research. Our focus is on empirical studies that either use LLMs as part of the research process or studies that evaluate existing or new tools that are based on LLMs. This paper contributes the first set of holistic guidelines for such studies. Our goal is to start a discussion in the software engineering research community to reach a common understanding of our standards for high-quality empirical studies involving LLMs. |
| title | Towards Evaluation Guidelines for Empirical Studies involving LLMs |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2411.07668 |