Cover Image

SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Adamenko, Pavel, Ivanov, Mikhail, Valeev, Aidar, Levichev, Rodion, Zadorozhny, Pavel, Lopatin, Ivan, Babayev, Dmitry, Fenogenova, Alena, Malykh, Valentin
Format:	Preprint
Published:	2025
Subjects:	Software Engineering Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

MERA Code: A Unified Framework for Evaluating Code Generation Across Tasks
by: Chervyakov, Artem, et al.
Published: (2025)

Exploring the Potential of Conversational Test Suite Based Program Repair on SWE-bench
by: Cheshkov, Anton, et al.
Published: (2024)

SWE-Sharp-Bench: A Reproducible Benchmark for C# Software Engineering Tasks
by: Mhatre, Sanket, et al.
Published: (2025)

YABLoCo: Yet Another Benchmark for Long Context Code Generation
by: Valeev, Aidar, et al.
Published: (2025)

SWE-Next: Scalable Real-World Software Engineering Tasks for Agents
by: Liang, Jiarong, et al.
Published: (2026)

Finetuning Large Language Models for Vulnerability Detection
by: Shestov, Alexey, et al.
Published: (2024)

Searching by Code: a New SearchBySnippet Dataset and SnippeR Retrieval Model for Searching by Code Snippets
by: Sedykh, Ivan, et al.
Published: (2023)

Iterative Self-Training for Code Generation via Reinforced Re-Ranking
by: Sorokin, Nikita, et al.
Published: (2025)

SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents
by: Badertdinov, Ibragim, et al.
Published: (2025)

SWE-Hub: A Unified Production System for Scalable, Executable Software Engineering Tasks
by: Zeng, Yucheng, et al.
Published: (2026)

Training Software Engineering Agents and Verifiers with SWE-Gym
by: Pan, Jiayi, et al.
Published: (2024)

SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
by: Deng, Xiang, et al.
Published: (2025)

TOM-SWE: User Mental Modeling For Software Engineering Agents
by: Zhou, Xuhui, et al.
Published: (2025)

StRuCom: A Novel Dataset of Structured Code Comments in Russian
by: Dziuba, Maria, et al.
Published: (2025)

CIDRe: A Reference-Free Multi-Aspect Criterion for Code Comment Quality Measurement
by: Dziuba, Maria, et al.
Published: (2025)

SWE-smith: Scaling Data for Software Engineering Agents
by: Yang, John, et al.
Published: (2025)

From SWE-ZERO to SWE-HERO: Execution-free to Execution-based Fine-tuning for Software Engineering Agents
by: Ludwig, Nikolai, et al.
Published: (2026)

The Software Engineering Simulations Lab: Agentic AI for RE Quality Simulations
by: Femmer, Henning, et al.
Published: (2025)

SWE-Arena: An Interactive Platform for Evaluating Foundation Models in Software Engineering
by: Zhao, Zhimin
Published: (2025)

SWE-World: Building Software Engineering Agents in Docker-Free Environments
by: Sun, Shuang, et al.
Published: (2026)

SWE-Master: Unleashing the Potential of Software Engineering Agents via Post-Training
by: Song, Huatong, et al.
Published: (2026)

GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agents
by: Shetty, Manish, et al.
Published: (2025)

CCT-Code: Cross-Consistency Training for Multilingual Clone Detection and Code Search
by: Tikhonov, Anton, et al.
Published: (2023)

SWE-Replay: Efficient Test-Time Scaling for Software Engineering Agents
by: Ding, Yifeng, et al.
Published: (2026)

SWE-Bench++: A Framework for the Scalable Generation of Software Engineering Benchmarks from Open-Source Repositories
by: Wang, Lilin, et al.
Published: (2025)

SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale
by: Badertdinov, Ibragim, et al.
Published: (2026)

SWE-Bench+: Enhanced Coding Benchmark for LLMs
by: Aleithan, Reem, et al.
Published: (2024)

Satori-SWE: Evolutionary Test-Time Scaling for Sample-Efficient Software Engineering
by: Zeng, Guangtao, et al.
Published: (2025)

Rethinking Software Engineering for Agentic AI Systems
by: Alenezi, Mamdouh
Published: (2026)

SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?
by: Han, Tingxu, et al.
Published: (2026)

Assessing and Advancing Benchmarks for Evaluating Large Language Models in Software Engineering Tasks
by: Hu, Xing, et al.
Published: (2025)

SWE-MiniSandbox: Container-Free Reinforcement Learning for Building Software Engineering Agents
by: Yuan, Danlong, et al.
Published: (2026)

A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic System
by: Guo, Jiale, et al.
Published: (2025)

Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?
by: Xia, Chunqiu Steven, et al.
Published: (2025)

SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?
by: Miserendino, Samuel, et al.
Published: (2025)

SWE-AGI: Benchmarking Specification-Driven Software Construction with MoonBit in the Era of Autonomous Agents
by: Zhang, Zhirui, et al.
Published: (2026)

Agentic AI in the Software Development Lifecycle: Architecture, Empirical Evidence, and the Reshaping of Software Engineering
by: Bhati, Happy
Published: (2026)

Agentic AI Software Engineers: Programming with Trust
by: Roychoudhury, Abhik, et al.
Published: (2025)

Agentic Pipelines in Embedded Software Engineering: Emerging Practices and Challenges
by: Sun, Simin, et al.
Published: (2026)

SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
by: Yang, John, et al.
Published: (2024)