Imagen de Portada

AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Sahoo, Priyam, Mittal, Gaurav, Li, Xiaomin, Ma, Shengjie, Steenhoek, Benjamin, Lin, Pingping, Hu, Yu
Formato:	Preprint
Publicado:	2026
Materias:	Software Engineering Artificial Intelligence
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluation
por: Garg, Spandan, et al.
Publicado: (2025)

Learning Correct Behavior from Examples: Validating Sequential Execution in Autonomous Agents
por: Sharma, Reshabh K, et al.
Publicado: (2026)

Kimi-Dev: Agentless Training as Skill Prior for SWE-Agents
por: Yang, Zonghan, et al.
Publicado: (2025)

When Agents go Astray: Course-Correcting SWE Agents with PRMs
por: Gandhi, Shubham, et al.
Publicado: (2025)

SWE-Effi: Re-Evaluating Software AI Agent System Effectiveness Under Resource Constraints
por: Fan, Zhiyu, et al.
Publicado: (2025)

SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration
por: Chen, Jialong, et al.
Publicado: (2026)

TOM-SWE: User Mental Modeling For Software Engineering Agents
por: Zhou, Xuhui, et al.
Publicado: (2025)

SWE-Next: Scalable Real-World Software Engineering Tasks for Agents
por: Liang, Jiarong, et al.
Publicado: (2026)

SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?
por: Han, Tingxu, et al.
Publicado: (2026)

SWE-Bench-CL: Continual Learning for Coding Agents
por: Joshi, Thomas, et al.
Publicado: (2025)

SWE-smith: Scaling Data for Software Engineering Agents
por: Yang, John, et al.
Publicado: (2025)

GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agents
por: Shetty, Manish, et al.
Publicado: (2025)

SWE-Fuse: Empowering Software Agents via Issue-free Trajectory Learning and Entropy-aware RLVR Training
por: Wen, Xin-Cheng, et al.
Publicado: (2026)

Resolving Java Code Repository Issues with iSWE Agent
por: Ganhotra, Jatin, et al.
Publicado: (2026)

SWE-chat: Coding Agent Interactions From Real Users in the Wild
por: Baumann, Joachim, et al.
Publicado: (2026)

SWE-Replay: Efficient Test-Time Scaling for Software Engineering Agents
por: Ding, Yifeng, et al.
Publicado: (2026)

SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades
por: Lam, Man Ho, et al.
Publicado: (2026)

SWE-AGI: Benchmarking Specification-Driven Software Construction with MoonBit in the Era of Autonomous Agents
por: Zhang, Zhirui, et al.
Publicado: (2026)

SWE-Cycle: Benchmarking Code Agents across the Complete Issue Resolution Cycle
por: Guan, Hao, et al.
Publicado: (2026)

SWE-Shepherd: Advancing PRMs for Reinforcing Code Agents
por: Dihan, Mahir Labib, et al.
Publicado: (2026)

SWE-TRACE: Optimizing Long-Horizon SWE Agents Through Rubric Process Reward Models and Heuristic Test-Time Scaling
por: Han, Hao, et al.
Publicado: (2026)

SWE-MiniSandbox: Container-Free Reinforcement Learning for Building Software Engineering Agents
por: Yuan, Danlong, et al.
Publicado: (2026)

Beyond Verifiable Rewards: Rubric-Based GRM for Reinforced Fine-Tuning SWE Agents
por: Huang, Jiawei, et al.
Publicado: (2026)

SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models
por: Xu, Jingxuan, et al.
Publicado: (2025)

CodeSense: a Real-World Benchmark and Dataset for Code Semantic Reasoning
por: Roy, Monoshi Kumar, et al.
Publicado: (2025)

Dissecting the SWE-Bench Leaderboards: Profiling Submitters and Architectures of LLM- and Agent-Based Repair Systems
por: Martinez, Matias, et al.
Publicado: (2025)

SWE-Universe: Scale Real-World Verifiable Environments to Millions
por: Chen, Mouxiang, et al.
Publicado: (2026)

GHIssuemarket: A Sandbox Environment for SWE-Agents Economic Experimentation
por: Fouad, Mohamed A., et al.
Publicado: (2024)

Understanding Automated Program Repair Agents Through the Lens of Traceability: An Empirical Study
por: Ceka, Ira, et al.
Publicado: (2025)

SWE-bench Goes Live!
por: Zhang, Linghao, et al.
Publicado: (2025)

SWE-Factory: Your Automated Factory for Issue Resolution Training Data and Evaluation Benchmarks
por: Guo, Lianghong, et al.
Publicado: (2025)

SWE Context Bench: A Benchmark for Context Learning in Coding
por: Zhu, Jiayuan, et al.
Publicado: (2026)

APEX-SWE
por: Kottamasu, Abhi, et al.
Publicado: (2026)

CodeMem: Architecting Reproducible Agents via Dynamic MCP and Procedural Memory
por: Gaurav, Nishant, et al.
Publicado: (2025)

BUILD-AND-FIND: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases
por: Lin, Jhen-Ke
Publicado: (2026)

UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench
por: Yu, Boxi, et al.
Publicado: (2025)

Does SWE-Bench-Verified Test Agent Ability or Model Memory?
por: Prathifkumar, Thanosan, et al.
Publicado: (2025)

OSS-UAgent: An Agent-based Usability Evaluation Framework for Open Source Software
por: Meng, Lingkai, et al.
Publicado: (2025)

Toward Training Superintelligent Software Agents through Self-Play SWE-RL
por: Wei, Yuxiang, et al.
Publicado: (2025)

Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?
por: Xia, Chunqiu Steven, et al.
Publicado: (2025)