Cover Image

What's in a Benchmark? The Case of SWE-Bench in Automated Program Repair

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Martinez, Matias, Franch, Xavier
Format:	Preprint
Published:	2026
Subjects:	Software Engineering
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Dissecting the SWE-Bench Leaderboards: Profiling Submitters and Architectures of LLM- and Agent-Based Repair Systems
by: Martinez, Matias, et al.
Published: (2025)

Energy Consumption of Automated Program Repair
by: Martinez, Matias, et al.
Published: (2022)

SWE-Bench+: Enhanced Coding Benchmark for LLMs
by: Aleithan, Reem, et al.
Published: (2024)

Automated Requirements Relation Extraction
by: Motger, Quim, et al.
Published: (2024)

SWE-Sharp-Bench: A Reproducible Benchmark for C# Software Engineering Tasks
by: Mhatre, Sanket, et al.
Published: (2025)

SWE Context Bench: A Benchmark for Context Learning in Coding
by: Zhu, Jiayuan, et al.
Published: (2026)

Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluation
by: Garg, Spandan, et al.
Published: (2025)

HEJ-Robust: A Robustness Benchmark for LLM-Based Automated Program Repair
by: Rabbi, Fazle, et al.
Published: (2026)

The Impact of Program Reduction on Automated Program Repair
by: Vidziunas, Linas, et al.
Published: (2024)

ThinkRepair: Self-Directed Automated Program Repair
by: Yin, Xin, et al.
Published: (2024)

CI-Repair-Bench: A Repository-Aware Benchmark for Automated Patch Validation via CI Workflows
by: Muna, Rabeya Khatun, et al.
Published: (2026)

ContrastRepair: Enhancing Conversation-Based Automated Program Repair via Contrastive Test Case Pairs
by: Kong, Jiaolong, et al.
Published: (2024)

Cataloguing Hugging Face Models to Software Engineering Activities: Automation and Findings
by: González, Alexandra, et al.
Published: (2025)

Specification Vibing for Automated Program Repair
by: Zhu, Taohong, et al.
Published: (2026)

Does SWE-Bench-Verified Test Agent Ability or Model Memory?
by: Prathifkumar, Thanosan, et al.
Published: (2025)

SEMODS: A Validated Dataset of Open-Source Software Engineering Models
by: González, Alexandra, et al.
Published: (2026)

SPICE: An Automated SWE-Bench Labeling Pipeline for Issue Clarity, Test Coverage, and Effort Estimation
by: Oliva, Gustavo A., et al.
Published: (2025)

RepairBench: Leaderboard of Frontier Models for Program Repair
by: Silva, André, et al.
Published: (2024)

Exploring the Potential of Conversational Test Suite Based Program Repair on SWE-bench
by: Cheshkov, Anton, et al.
Published: (2024)

SWE-Factory: Your Automated Factory for Issue Resolution Training Data and Evaluation Benchmarks
by: Guo, Lianghong, et al.
Published: (2025)

PathFix: Automated Program Repair with Expected Path
by: He, Xu, et al.
Published: (2025)

On The Effectiveness of Dynamic Reduction Techniques in Automated Program Repair
by: Al-Bataineh, Omar I.
Published: (2024)

Towards Practical and Useful Automated Program Repair for Debugging
by: Xin, Qi, et al.
Published: (2024)

BUGSPHP: A dataset for Automated Program Repair in PHP
by: Pramod, K. D., et al.
Published: (2024)

ASAP-Repair: API-Specific Automated Program Repair Based on API Usage Graphs
by: Nielebock, Sebastian, et al.
Published: (2024)

Characterizing Datasets for LLM-based Requirements Engineering: A Systematic Mapping Study
by: Motger, Quim, et al.
Published: (2025)

SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents
by: Rashid, Muhammad Shihab, et al.
Published: (2025)

Hybrid Automated Program Repair by Combining Large Language Models and Program Analysis
by: Li, Fengjie, et al.
Published: (2024)

A Tool for Automatically Cataloguing and Selecting Pre-Trained Models and Datasets for Software Engineering
by: González, Alexandra, et al.
Published: (2026)

Automated Repair of C Programs Using Large Language Models
by: Farzandway, Mahdi, et al.
Published: (2025)

Exploring Generalizable Automated Program Repair with Large Language Models
by: Campos, Viola, et al.
Published: (2025)

RESTORE: Retrospective Fault Localization Enhancing Automated Program Repair
by: Xu, Tongtong, et al.
Published: (2019)

Adapting Knowledge Prompt Tuning for Enhanced Automated Program Repair
by: Cai, Xuemeng, et al.
Published: (2025)

RePurr: Automated Repair of Block-Based Learners' Programs
by: Schweikl, Sebastian, et al.
Published: (2025)

Empirical Evaluation of Large Language Models in Automated Program Repair
by: Sun, Jiajun, et al.
Published: (2025)

TSAPR: A Tree Search Framework For Automated Program Repair
by: Hu, Haichuan, et al.
Published: (2025)

Reinforcement Learning for Mutation Operator Selection in Automated Program Repair
by: Hanna, Carol, et al.
Published: (2023)

Automated Test Case Repair Using Language Models
by: Yaraghi, Ahmadreza Saboor, et al.
Published: (2024)

DebugRepair: Enhancing LLM-Based Automated Program Repair via Self-Directed Debugging
by: Wu, Linhao, et al.
Published: (2026)

SWE-Bench Mobile: Can Large Language Model Agents Develop Industry-Level Mobile Applications?
by: Tian, Muxin, et al.
Published: (2026)