Cover Image

Assessing and Advancing Benchmarks for Evaluating Large Language Models in Software Engineering Tasks

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Hu, Xing, Niu, Feifei, Chen, Junkai, Zhou, Xin, Zhang, Junwei, He, Junda, Xia, Xin, Lo, David
Format:	Preprint
Published:	2025
Subjects:	Software Engineering
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

An Empirical Study of Speculative Decoding on Software Engineering Tasks
by: Li, Yijia, et al.
Published: (2026)

The Current Challenges of Software Engineering in the Era of Large Language Models
by: Gao, Cuiyun, et al.
Published: (2024)

LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead
by: He, Junda, et al.
Published: (2024)

Towards Understanding Bugs in Distributed Training and Inference Frameworks for Large Language Models
by: Yu, Xiao, et al.
Published: (2025)

When Deep Learning Meets Information Retrieval-based Bug Localization: A Survey
by: Niu, Feifei, et al.
Published: (2025)

Efficient and Green Large Language Models for Software Engineering: Literature Review, Vision, and the Road Ahead
by: Shi, Jieke, et al.
Published: (2024)

Greening Large Language Models of Code
by: Shi, Jieke, et al.
Published: (2023)

LLM-as-a-Judge for Software Engineering: Literature Review, Vision, and the Road Ahead
by: He, Junda, et al.
Published: (2025)

Identifying and Mitigating API Misuse in Large Language Models
by: Zhuo, Terry Yue, et al.
Published: (2025)

Large Language Model for Vulnerability Detection: Emerging Results and Future Directions
by: Zhou, Xin, et al.
Published: (2024)

NLPerturbator: Studying the Robustness of Code LLMs to Natural Language Variations
by: Chen, Junkai, et al.
Published: (2024)

Feature Request Analysis and Processing: Tasks, Techniques, and Trends
by: Niu, Feifei, et al.
Published: (2025)

Revisiting Sentiment Analysis for Software Engineering in the Era of Large Language Models
by: Zhang, Ting, et al.
Published: (2023)

Large Language Model-Based Agents for Software Engineering: A Survey
by: Liu, Junwei, et al.
Published: (2024)

LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks
by: Zhou, Xin, et al.
Published: (2025)

A Benchmark for Evaluating Repository-Level Code Agents with Intermediate Reasoning on Feature Addition Task
by: Liu, Shuhan, et al.
Published: (2026)

A Functional Software Reference Architecture for LLM-Integrated Systems
by: Bucaioni, Alessio, et al.
Published: (2025)

Artificial Intelligence for Software Architecture: Literature Review and the Road Ahead
by: Bucaioni, Alessio, et al.
Published: (2025)

Open the Oyster: Empirical Evaluation and Improvement of Code Reasoning Confidence in LLMs
by: Wang, Shufan, et al.
Published: (2025)

Large Language Model for Vulnerability Detection and Repair: Literature Review and the Road Ahead
by: Zhou, Xin, et al.
Published: (2024)

Towards an Understanding of Large Language Models in Software Engineering Tasks
by: Zheng, Zibin, et al.
Published: (2023)

Less is More: On the Importance of Data Quality for Unit Test Generation
by: Zhang, Junwei, et al.
Published: (2025)

Benchmarking Large Language Models for Multi-Language Software Vulnerability Detection
by: Zhang, Ting, et al.
Published: (2025)

From Code to Courtroom: LLMs as the New Software Judges
by: He, Junda, et al.
Published: (2025)

A Software Engineering Perspective on Testing Large Language Models: Research, Practice, Tools and Benchmarks
by: Hudson, Sinclair, et al.
Published: (2024)

Verify Implementation Equivalence of Large Models
by: Zhan, Qi, et al.
Published: (2026)

Where Is Self-admitted Code Generated by Large Language Models on GitHub?
by: Yu, Xiao, et al.
Published: (2024)

Hotfixing Large Language Models for Code
by: Yang, Zhou, et al.
Published: (2024)

Guidelines for Empirical Studies in Software Engineering involving Large Language Models
by: Baltes, Sebastian, et al.
Published: (2025)

Towards Explainable Vulnerability Detection with Large Language Models
by: Mao, Qiheng, et al.
Published: (2024)

Every Maintenance Has Its Exemplar: The Future of Software Maintenance through Migration
by: Chen, Zirui, et al.
Published: (2026)

CREME: Robustness Enhancement of Code LLMs via Layer-Aware Model Editing
by: Liu, Shuhan, et al.
Published: (2025)

Enhancement Report Approval Prediction: A Comparative Study of Large Language Models
by: Zuo, Haosheng, et al.
Published: (2025)

Re-Evaluating Code LLM Benchmarks Under Semantic Mutation
by: Pan, Zhiyuan, et al.
Published: (2025)

FeedbackEval: A Benchmark for Evaluating Large Language Models in Feedback-Driven Code Repair Tasks
by: Dai, Dekun, et al.
Published: (2025)

Bridging Bug Localization and Issue Fixing: A Hierarchical Localization Framework Leveraging Large Language Models
by: Chang, Jianming, et al.
Published: (2025)

Large Language Models for Software Engineering: A Systematic Literature Review
by: Hou, Xinyi, et al.
Published: (2023)

Understanding Practitioners' Expectations on Clear Code Review Comments
by: Chen, Junkai, et al.
Published: (2024)

APIDocBooster: An Extract-Then-Abstract Framework Leveraging Large Language Models for Augmenting API Documentation
by: Yang, Chengran, et al.
Published: (2023)

SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks
by: Adamenko, Pavel, et al.
Published: (2025)