Cover Image

Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studies

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Angermeir, Florian, Amougou, Maximilian, Kreitz, Mark, Bauer, Andreas, Linhuber, Matthias, Fucci, Davide, C., Fabiola Moyón, Mendez, Daniel, Gorschek, Tony
Format:	Preprint
Published:	2025
Subjects:	Software Engineering Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Aligning Security Compliance and DevOps: A Longitudinal Study
by: Moyón, Fabiola, et al.
Published: (2025)

Industrial Challenges in Secure Continuous Development
by: Moyón, Fabiola, et al.
Published: (2024)

Enterprise-Driven Open Source Software: A Case Study on Security Automation
by: Angermeir, Florian, et al.
Published: (2021)

Towards Automated Continuous Security Compliance
by: Angermeir, Florian, et al.
Published: (2024)

An Agentic Approach Towards Replication Package Quality Evaluation
by: Mbida, Maximilian Alexander Amougou, et al.
Published: (2026)

On Developing an Artifact-based Approach to Regulatory Requirements Engineering
by: Kosenkov, Oleksandr, et al.
Published: (2024)

No Free Lunch: Research Software Testing in Teaching
by: Dorner, Michael, et al.
Published: (2024)

Automated Security Findings Management: A Case Study in Industrial DevOps
by: Voggenreiter, Markus, et al.
Published: (2024)

Systematic Mapping Study on Requirements Engineering for Regulatory Compliance of Software Systems
by: Kosenkov, Oleksandr, et al.
Published: (2024)

Experience Report on the Adaptable Integration of Requirements Engineering Courses into Curricula for Professionals
by: Kosenkov, Oleksandr, et al.
Published: (2026)

Policy-driven Software Bill of Materials on GitHub: An Empirical Study
by: Novikov, Oleksii, et al.
Published: (2025)

Privacy by Design: Aligning GDPR and Software Engineering Specifications with a Requirements Engineering Approach
by: Kosenkov, Oleksandr, et al.
Published: (2025)

Replications, Revisions, and Reanalyses: Managing Variance Theories in Software Engineering
by: Frattini, Julian, et al.
Published: (2024)

Guidelines for Empirical Studies in Software Engineering involving Large Language Models
by: Baltes, Sebastian, et al.
Published: (2025)

Crossover Designs in Software Engineering Experiments: Review of the State of Analysis
by: Frattini, Julian, et al.
Published: (2024)

Towards a Goal-Centric Assessment of Requirements Engineering Methods for Privacy by Design
by: Kosenkov, Oleksandr, et al.
Published: (2026)

Dealing with SonarQube Cloud: Initial Results from a Mining Software Repository Study
by: Nocera, Sabato, et al.
Published: (2025)

Augmenting Software Bills of Materials with Software Vulnerability Description: A Preliminary Study on GitHub
by: Fucci, Davide, et al.
Published: (2025)

An Empirical Study of Generative AI Adoption in Software Engineering
by: Giray, Görkem, et al.
Published: (2025)

A Second Look at the Impact of Passive Voice Requirements on Domain Modeling: Bayesian Reanalysis of an Experiment
by: Frattini, Julian, et al.
Published: (2024)

NLP4RE Tools: Classification, Overview, and Management
by: Frattini, Julian, et al.
Published: (2024)

From Requirements to Code: Understanding Developer Practices in LLM-Assisted Software Engineering
by: Ullrich, Jonathan, et al.
Published: (2025)

Teaching Empirical Research Methods in Software Engineering: An Editorial Introduction
by: Mendez, Daniel, et al.
Published: (2025)

Designing a Syllabus for a Course on Empirical Software Engineering
by: Avgeriou, Paris, et al.
Published: (2025)

Measuring the Fitness-for-Purpose of Requirements: An initial Model of Activities and Attributes
by: Frattini, Julian, et al.
Published: (2024)

Empirical and Sustainability Aspects of Software Engineering Research in the Era of Large Language Models: A Reflection
by: Williams, David, et al.
Published: (2025)

Assured LLM-Based Software Engineering
by: Alshahwan, Nadia, et al.
Published: (2024)

LLM-Assisted Empirical Software Engineering: Systematic Literature Review and Research Agenda
by: Gomes, Victoria, et al.
Published: (2026)

An Empirical study on LLM-based Log Retrieval for Software Engineering Metadata Management
by: Sun, Simin, et al.
Published: (2025)

Taxing Collaborative Software Engineering
by: Dorner, Michael, et al.
Published: (2023)

Replication in Requirements Engineering: the NLP for RE Case
by: Abualhaija, Sallam, et al.
Published: (2023)

Autonomous LLM-generated Feedback for Student Exercises in Introductory Software Engineering Courses
by: Metzger, Andreas
Published: (2026)

An Empirical Investigation of the Experiences of Dyslexic Software Engineers
by: Cruz, Marcos Vinicius, et al.
Published: (2025)

On the Replicability and Reproducibility of Deep Learning in Software Engineering
by: Liu, Chao, et al.
Published: (2020)

Requirements Quality Research Artifacts: Recovery, Analysis, and Management Guideline
by: Frattini, Julian, et al.
Published: (2024)

A Live Extensible Ontology of Quality Factors for Textual Requirements
by: Frattini, Julian, et al.
Published: (2022)

OLAF: Towards Robust LLM-Based Annotation Framework in Empirical Software Engineering
by: Imran, Mia Mohammad, et al.
Published: (2025)

SWE-Sharp-Bench: A Reproducible Benchmark for C# Software Engineering Tasks
by: Mhatre, Sanket, et al.
Published: (2025)

Large Language Models for Software Engineering: A Reproducibility Crisis
by: Siddiq, Mohammed Latif, et al.
Published: (2025)

Reproducible, Explainable, and Effective Evaluations of Agentic AI for Software Engineering
by: Li, Jingyue, et al.
Published: (2026)