Investigating the Use of LLMs for Evidence Briefings Generation in Software Engineering
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911068063793152 |
|---|---|
| author | Marcelino, Mauro Alves, Marcos Trinkenreich, Bianca Cartaxo, Bruno Soares, Sérgio Barbosa, Simone D. J. Kalinowski, Marcos |
| author_facet | Marcelino, Mauro Alves, Marcos Trinkenreich, Bianca Cartaxo, Bruno Soares, Sérgio Barbosa, Simone D. J. Kalinowski, Marcos |
| contents | [Context] An evidence briefing is a concise and objective transfer medium that can present the main findings of a study to software engineers in the industry. Although practitioners and researchers have deemed Evidence Briefings useful, their production requires manual labor, which may be a significant challenge to their broad adoption. [Goal] The goal of this registered report is to describe an experimental protocol for evaluating LLM-generated evidence briefings for secondary studies in terms of content fidelity, ease of understanding, and usefulness, as perceived by researchers and practitioners, compared to human-made briefings. [Method] We developed an RAG-based LLM tool to generate evidence briefings. We used the tool to automatically generate two evidence briefings that had been manually generated in previous research efforts. We designed a controlled experiment to evaluate how the LLM-generated briefings compare to the human-made ones regarding perceived content fidelity, ease of understanding, and usefulness. [Results] To be reported after the experimental trials. [Conclusion] Depending on the experiment results. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_15828 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Investigating the Use of LLMs for Evidence Briefings Generation in Software Engineering Marcelino, Mauro Alves, Marcos Trinkenreich, Bianca Cartaxo, Bruno Soares, Sérgio Barbosa, Simone D. J. Kalinowski, Marcos Software Engineering [Context] An evidence briefing is a concise and objective transfer medium that can present the main findings of a study to software engineers in the industry. Although practitioners and researchers have deemed Evidence Briefings useful, their production requires manual labor, which may be a significant challenge to their broad adoption. [Goal] The goal of this registered report is to describe an experimental protocol for evaluating LLM-generated evidence briefings for secondary studies in terms of content fidelity, ease of understanding, and usefulness, as perceived by researchers and practitioners, compared to human-made briefings. [Method] We developed an RAG-based LLM tool to generate evidence briefings. We used the tool to automatically generate two evidence briefings that had been manually generated in previous research efforts. We designed a controlled experiment to evaluate how the LLM-generated briefings compare to the human-made ones regarding perceived content fidelity, ease of understanding, and usefulness. [Results] To be reported after the experimental trials. [Conclusion] Depending on the experiment results. |
| title | Investigating the Use of LLMs for Evidence Briefings Generation in Software Engineering |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2507.15828 |