Imagen de Portada

Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Wang, Ruiqi, Guo, Jiyu, Gao, Cuiyun, Fan, Guodong, Chong, Chun Yong, Xia, Xin
Formato:	Preprint
Publicado:	2025
Materias:	Software Engineering Artificial Intelligence
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

AXIOM: Benchmarking LLM-as-a-Judge for Code via Rule-Based Perturbation and Multisource Quality Calibration
por: Wang, Ruiqi, et al.
Publicado: (2025)

An Empirical Study of Knowledge Distillation for Code Understanding Tasks
por: Wang, Ruiqi, et al.
Publicado: (2025)

The Current Challenges of Software Engineering in the Era of Large Language Models
por: Gao, Cuiyun, et al.
Publicado: (2024)

A Systematic Literature Review of Code Hallucinations in LLMs: Characterization, Mitigation Methods, Challenges, and Future Directions for Reliable AI
por: Gao, Cuiyun, et al.
Publicado: (2025)

ComplexCodeEval: A Benchmark for Evaluating Large Code Models on More Complex Code
por: Feng, Jia, et al.
Publicado: (2024)

When Model Editing Meets Service Evolution: A Knowledge-Update Perspective for Service Recommendation
por: Fan, Guodong, et al.
Publicado: (2026)

An Empirical Study of Speculative Decoding on Software Engineering Tasks
por: Li, Yijia, et al.
Publicado: (2026)

An Empirical Study of Retrieval-Augmented Code Generation: Challenges and Opportunities
por: Yang, Zezhou, et al.
Publicado: (2025)

A Systematic Evaluation of Large Code Models in API Suggestion: When, Which, and How
por: Wang, Chaozheng, et al.
Publicado: (2024)

On the Replicability and Reproducibility of Deep Learning in Software Engineering
por: Liu, Chao, et al.
Publicado: (2020)

Can LLMs Replace Manual Annotation of Software Engineering Artifacts?
por: Ahmed, Toufique, et al.
Publicado: (2024)

A Comprehensive Empirical Evaluation of Agent Frameworks on Code-centric Software Engineering Tasks
por: Yin, Zhuowen, et al.
Publicado: (2025)

On the Effectiveness of Context Compression for Repository-Level Tasks: An Empirical Investigation
por: Feng, Jia, et al.
Publicado: (2026)

SR-Eval: Evaluating LLMs on Code Generation under Stepwise Requirement Refinement
por: Zhan, Zexun, et al.
Publicado: (2025)

A Framework for Using LLMs for Repository Mining Studies in Empirical Software Engineering
por: de Martino, Vincenzo, et al.
Publicado: (2024)

Game Rewards Vulnerabilities: Software Vulnerability Detection with Zero-Sum Game and Prototype Learning
por: Wen, Xin-Cheng, et al.
Publicado: (2024)

Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios
por: Hu, Ruida, et al.
Publicado: (2026)

Can LLMs be Effective Code Contributors? A Study on Open-source Projects
por: Chong, Chun Jie, et al.
Publicado: (2026)

LLM-as-a-Judge for Software Engineering: Literature Review, Vision, and the Road Ahead
por: He, Junda, et al.
Publicado: (2025)

Can LLMs Replace Humans During Code Chunking?
por: Glasz, Christopher, et al.
Publicado: (2025)

EvalSVA: Multi-Agent Evaluators for Next-Gen Software Vulnerability Assessment
por: Wen, Xin-Cheng, et al.
Publicado: (2024)

An Empirical Study on the Potential of LLMs in Automated Software Refactoring
por: Liu, Bo, et al.
Publicado: (2024)

An Empirical Study of Generative AI Adoption in Software Engineering
por: Giray, Görkem, et al.
Publicado: (2025)

Code Digital Twin: Empowering LLMs with Tacit Knowledge for Complex Software Development
por: Peng, Xin, et al.
Publicado: (2025)

The Prompt Alchemist: Automated LLM-Tailored Prompt Optimization for Test Case Generation
por: Gao, Shuzheng, et al.
Publicado: (2025)

An Empirical Study of Perceptions of General LLMs and Multimodal LLMs on Hugging Face
por: Liu, Yujian, et al.
Publicado: (2026)

Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studies
por: Angermeir, Florian, et al.
Publicado: (2025)

An Empirical Study on Challenges for LLM Application Developers
por: Chen, Xiang, et al.
Publicado: (2024)

Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineering
por: Zhao, Zixiao, et al.
Publicado: (2026)

VulEval: Towards Repository-Level Evaluation of Software Vulnerability Detection
por: Wen, Xin-Cheng, et al.
Publicado: (2024)

Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scaling
por: Trae Research Team, et al.
Publicado: (2025)

Less is More? An Empirical Study on Configuration Issues in Python PyPI Ecosystem
por: Peng, Yun, et al.
Publicado: (2023)

Can GPT-4 Replicate Empirical Software Engineering Research?
por: Liang, Jenny T., et al.
Publicado: (2023)

Evaluating Repository-level Software Documentation via Question Answering and Feature-Driven Development
por: Wang, Xinchen, et al.
Publicado: (2026)

From Code to Courtroom: LLMs as the New Software Judges
por: He, Junda, et al.
Publicado: (2025)

LLM-Assisted Empirical Software Engineering: Systematic Literature Review and Research Agenda
por: Gomes, Victoria, et al.
Publicado: (2026)

An Empirical study on LLM-based Log Retrieval for Software Engineering Metadata Management
por: Sun, Simin, et al.
Publicado: (2025)

Integrating Rules and Semantics for LLM-Based C-to-Rust Translation
por: Luo, Feng, et al.
Publicado: (2025)

An LLM-as-Judge Metric for Bridging the Gap with Human Evaluation in SE Tasks
por: Zhou, Xin, et al.
Publicado: (2025)

Guidelines for Empirical Studies in Software Engineering involving Large Language Models
por: Baltes, Sebastian, et al.
Publicado: (2025)