Image de couverture de livre

SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Bogin, Ben, Yang, Kejuan, Gupta, Shashank, Richardson, Kyle, Bransom, Erin, Clark, Peter, Sabharwal, Ashish, Khot, Tushar
Format:	Preprint
Publié:	2024
Sujets:	Artificial Intelligence Computation and Language Software Engineering
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents
par: Trivedi, Harsh, et autres
Publié: (2024)

Leveraging In-Context Learning for Language Model Agents
par: Gupta, Shivanshu, et autres
Publié: (2025)

A Benchmark for Evaluating Repository-Level Code Agents with Intermediate Reasoning on Feature Addition Task
par: Liu, Shuhan, et autres
Publié: (2026)

Agentic Repository Mining: A Multi-Task Evaluation
par: Härtel, Johannes
Publié: (2026)

Evaluating and Improving Automated Repository-Level Rust Issue Resolution with LLM-based Agents
par: Xiang, Jiahong, et autres
Publié: (2026)

MOOT: a Repository of Many Multi-Objective Optimization Tasks
par: Menzies, Tim, et autres
Publié: (2025)

On the Effectiveness of Context Compression for Repository-Level Tasks: An Empirical Investigation
par: Feng, Jia, et autres
Publié: (2026)

TypeScript Repository Indexing for Code Agent Retrieval
par: Pu, Junsong, et autres
Publié: (2026)

Agent-Diff: Benchmarking LLM Agents on Enterprise API Tasks via Code Execution with State-Diff-Based Evaluation
par: Pysklo, Hubert M., et autres
Publié: (2026)

MutaGReP: Execution-Free Repository-Grounded Plan Search for Code-Use
par: Khan, Zaid, et autres
Publié: (2025)

Toward Executable Repository-Level Code Generation via Environment Alignment
par: Pan, Ruwei, et autres
Publié: (2026)

RepoTransAgent: Multi-Agent LLM Framework for Repository-Aware Code Translation
par: Guan, Ziqi, et autres
Publié: (2025)

GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging
par: Ni, Ziyi, et autres
Publié: (2025)

Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?
par: Gloaguen, Thibaud, et autres
Publié: (2026)

TimeMachine-bench: A Benchmark for Evaluating Model Capabilities in Repository-Level Migration Tasks
par: Fujii, Ryo, et autres
Publié: (2026)

Evaluating LLM Agents on Automated Software Analysis Tasks
par: Bouzenia, Islem, et autres
Publié: (2026)

Automated Benchmark Generation for Repository-Level Coding Tasks
par: Vergopoulos, Konstantinos, et autres
Publié: (2025)

Cross-Task Benchmarking and Evaluation of General-Purpose and Code-Specific Large Language Models
par: Das, Gunjan, et autres
Publié: (2025)

RepoDebug: Repository-Level Multi-Task and Multi-Language Debugging Evaluation of Large Language Models
par: Liu, Jingjing, et autres
Publié: (2025)

Leveraging Code to Improve In-context Learning for Semantic Parsing
par: Bogin, Ben, et autres
Publié: (2023)

FIKA: Expanding Dependency Reachability with Executability Guarantees
par: Gamage, Yogya, et autres
Publié: (2026)

MRG-Bench: Evaluating and Exploring the Requirements of Context for Repository-Level Code Generation
par: Li, Haiyang
Publié: (2025)

ResearchEnvBench: Benchmarking Agents on Environment Synthesis for Research Code Execution
par: Wang, Yubang, et autres
Publié: (2026)

SGAgent: Suggestion-Guided LLM-Based Multi-Agent Framework for Repository-Level Software Repair
par: Zhang, Quanjun, et autres
Publié: (2026)

A Task-Level Evaluation of AI Agents in Open-Source Projects
par: Rahman, Shojibur, et autres
Publié: (2026)

RepoMod-Bench: A Benchmark for Code Repository Modernization via Implementation-Agnostic Testing
par: Li, Xuefeng, et autres
Publié: (2026)

Teaching Mining Software Repositories
par: Codabux, Zadia, et autres
Publié: (2025)

HumanEvo: An Evolution-aware Benchmark for More Realistic Evaluation of Repository-level Code Generation
par: Zheng, Dewu, et autres
Publié: (2024)

Class-Level Code Generation from Natural Language Using Iterative, Tool-Enhanced Reasoning over Repository
par: Deshpande, Ajinkya, et autres
Publié: (2024)

Improving the Learning of Code Review Successive Tasks with Cross-Task Knowledge Distillation
par: Sghaier, Oussama Ben, et autres
Publié: (2024)

KAIJU: An Executive Kernel for Intent-Gated Execution of LLM Agents
par: Guerin, Cormac, et autres
Publié: (2026)

On the Creation of Representative Samples of Software Repositories
par: Gorostidi, June, et autres
Publié: (2024)

Modeling Sampling Workflows for Code Repositories
par: Lefeuvre, Romain, et autres
Publié: (2026)

Scaling Mobile Chaos Testing with AI-Driven Test Execution
par: Marcano, Juan, et autres
Publié: (2026)

A Comprehensive Empirical Evaluation of Agent Frameworks on Code-centric Software Engineering Tasks
par: Yin, Zhuowen, et autres
Publié: (2025)

CodeScore: Evaluating Code Generation by Learning Code Execution
par: Dong, Yihong, et autres
Publié: (2023)

TRACE: Evaluating Execution Efficiency of LLM-Based Code Translation
par: Gong, Zhihao, et autres
Publié: (2026)

TRACE: Evaluating Execution Efficiency of LLM-Based Code Translation
par: Gong, Zhihao, et autres
Publié: (2025)

CSR-Bench: Benchmarking LLM Agents in Deployment of Computer Science Research Repositories
par: Xiao, Yijia, et autres
Publié: (2025)

Analyzing the Evolution and Maintenance of Quantum Software Repositories
par: Upadhyay, Krishna, et autres
Publié: (2025)