Cover Image

A Real-World Benchmark for Evaluating Fine-Grained Issue Solving Capabilities of Large Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Hu, Ruida, Peng, Chao, Ren, Jingyi, Jiang, Bo, Meng, Xiangxin, Wu, Qinyun, Gao, Pengfei, Wang, Xinchen, Gao, Cuiyun
Format:	Preprint
Published:	2024
Subjects:	Software Engineering
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

CodeRepoQA: A Large-scale Benchmark for Software Engineering Question Answering
by: Hu, Ruida, et al.
Published: (2024)

AEGIS: An Agent-based Framework for General Bug Reproduction from Issue Descriptions
by: Wang, Xinchen, et al.
Published: (2024)

Benchmarking LLMs for Fine-Grained Code Review with Enriched Context in Practice
by: Hu, Ruida, et al.
Published: (2025)

CodeVisionary: An Agent-based Framework for Evaluating Large Language Models in Code Generation
by: Wang, Xinchen, et al.
Published: (2025)

Evaluating Repository-level Software Documentation via Question Answering and Feature-Driven Development
by: Wang, Xinchen, et al.
Published: (2026)

RepoMasterEval: Evaluating Code Completion via Real-World Repositories
by: Wu, Qinyun, et al.
Published: (2024)

Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios
by: Hu, Ruida, et al.
Published: (2026)

Repo2Run: Automated Building Executable Environment for Code Repository at Scale
by: Hu, Ruida, et al.
Published: (2025)

SR-Eval: Evaluating LLMs on Code Generation under Stepwise Requirement Refinement
by: Zhan, Zexun, et al.
Published: (2025)

SoRFT: Issue Resolving with Subtask-oriented Reinforced Fine-Tuning
by: Ma, Zexiong, et al.
Published: (2025)

VulEval: Towards Repository-Level Evaluation of Software Vulnerability Detection
by: Wen, Xin-Cheng, et al.
Published: (2024)

An Empirical Study on LLM-based Agents for Automated Bug Fixing
by: Meng, Xiangxin, et al.
Published: (2024)

Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scaling
by: Trae Research Team, et al.
Published: (2025)

ReposVul: A Repository-Level High-Quality Vulnerability Dataset
by: Wang, Xinchen, et al.
Published: (2024)

Less is More? An Empirical Study on Configuration Issues in Python PyPI Ecosystem
by: Peng, Yun, et al.
Published: (2023)

ComplexCodeEval: A Benchmark for Evaluating Large Code Models on More Complex Code
by: Feng, Jia, et al.
Published: (2024)

AXIOM: Benchmarking LLM-as-a-Judge for Code via Rule-Based Perturbation and Multisource Quality Calibration
by: Wang, Ruiqi, et al.
Published: (2025)

CoRE: A Fine-Grained Code Reasoning Benchmark Beyond Output Prediction
by: Gao, Jun, et al.
Published: (2026)

DecompileBench: A Comprehensive Benchmark for Evaluating Decompilers in Real-World Scenarios
by: Gao, Zeyu, et al.
Published: (2025)

A Systematic Evaluation of Large Code Models in API Suggestion: When, Which, and How
by: Wang, Chaozheng, et al.
Published: (2024)

RAG or Fine-tuning? A Comparative Study on LCMs-based Code Completion in Industry
by: Wang, Chaozheng, et al.
Published: (2025)

The Current Challenges of Software Engineering in the Era of Large Language Models
by: Gao, Cuiyun, et al.
Published: (2024)

More with Less: An Empirical Study of Turn-Control Strategies for Efficient Coding Agents
by: Gao, Pengfei, et al.
Published: (2025)

Game Rewards Vulnerabilities: Software Vulnerability Detection with Zero-Sum Game and Prototype Learning
by: Wen, Xin-Cheng, et al.
Published: (2024)

MarsCode Agent: AI-native Automated Bug Fixing
by: Liu, Yizhou, et al.
Published: (2024)

DialogAgent: An Auto-engagement Agent for Code Question Answering Data Production
by: Liang, Xiaoyun, et al.
Published: (2024)

When Shared Worlds Break: Demystifying Defects in Multi-User Extended Reality Software Systems
by: Li, Shuqing, et al.
Published: (2025)

SEER: Enhancing Chain-of-Thought Code Generation through Self-Exploring Deep Reasoning
by: Gao, Shuzheng, et al.
Published: (2025)

PATCHEVAL: A New Benchmark for Evaluating LLMs on Patching Real-World Vulnerabilities
by: Wei, Zichao, et al.
Published: (2025)

Empirical Study of Code Large Language Models for Binary Security Patch Detection
by: Li, Qingyuan, et al.
Published: (2025)

GraphLocator: Graph-guided Causal Reasoning for Issue Localization
by: Liu, Wei, et al.
Published: (2025)

SWE-Fuse: Empowering Software Agents via Issue-free Trajectory Learning and Entropy-aware RLVR Training
by: Wen, Xin-Cheng, et al.
Published: (2026)

GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging
by: Ni, Ziyi, et al.
Published: (2025)

Cascaded Code Editing: Large-Small Model Collaboration for Effective and Efficient Code Editing
by: Wang, Chaozheng, et al.
Published: (2026)

Reducing Cost of LLM Agents with Trajectory Reduction
by: Xiao, Yuan-An, et al.
Published: (2025)

A Deep Dive into Retrieval-Augmented Generation for Code Completion: Experience on WeChat
by: Yang, Zezhou, et al.
Published: (2025)

Search-Based LLMs for Code Optimization
by: Gao, Shuzheng, et al.
Published: (2024)

EvalSVA: Multi-Agent Evaluators for Next-Gen Software Vulnerability Assessment
by: Wen, Xin-Cheng, et al.
Published: (2024)

Towards Mitigating API Hallucination in Code Generated by LLMs with Hierarchical Dependency Aware
by: Chen, Yujia, et al.
Published: (2025)

Benchmarking Multimodal LLMs on Code Generation for Complex Interactive Webpages
by: Wu, Fan, et al.
Published: (2026)