Imagen de Portada

BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Tu, Xinming, Wang, Tianze, Yingzhou, Lu, Huang, Kexin, Qu, Yuanhao, Mostafavi, Sara
Formato:	Preprint
Publicado:	2026
Materias:	Computation and Language Artificial Intelligence Software Engineering
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Benchmarking Open-Source Safety Guard Models: A Comprehensive Evaluation
por: Harsh, Reetu Raj, et al.
Publicado: (2026)

Guarded Repair for Harm-Aware Post-hoc Replacement of LLM Mathematical Reasoning
por: Xia, Haizhou
Publicado: (2026)

AInsteinBench: Benchmarking Coding Agents on Scientific Repositories
por: Duston, Titouan, et al.
Publicado: (2025)

BuildBench: Benchmarking LLM Agents on Compiling Real-World Open-Source Software
por: Zhang, Zehua, et al.
Publicado: (2025)

ProbGuard: Probabilistic Runtime Monitoring for LLM Agent Safety
por: Wang, Haoyu, et al.
Publicado: (2025)

AgentGuard: Runtime Verification of AI Agents
por: Koohestani, Roham
Publicado: (2025)

CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks
por: Jiang, Hongchao, et al.
Publicado: (2025)

UA-Code-Bench: A Competitive Programming Benchmark for Evaluating LLM Code Generation in Ukrainian
por: Syromiatnikov, Mykyta, et al.
Publicado: (2025)

OmniCode: A Benchmark for Evaluating Software Engineering Agents
por: Sonwane, Atharv, et al.
Publicado: (2026)

BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity
por: Diddee, Harshita, et al.
Publicado: (2026)

SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks
por: Orlanski, Gabriel, et al.
Publicado: (2026)

FlakyGuard: Automatically Fixing Flaky Tests at Industry Scale
por: Li, Chengpeng, et al.
Publicado: (2025)

ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development
por: Yang, Jie, et al.
Publicado: (2026)

Collu-Bench: A Benchmark for Predicting Language Model Hallucinations in Code
por: Jiang, Nan, et al.
Publicado: (2024)

LoCoBench-Agent: An Interactive Benchmark for LLM Agents in Long-Context Software Engineering
por: Qiu, Jielin, et al.
Publicado: (2025)

PrimeGuard: Safe and Helpful LLMs through Tuning-Free Routing
por: Manczak, Blazej, et al.
Publicado: (2024)

DataGovBench: Benchmarking LLM Agents for Real-World Data Governance Workflows
por: Liu, Zhou, et al.
Publicado: (2025)

Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluation
por: Garg, Spandan, et al.
Publicado: (2025)

CangjieBench: Benchmarking LLMs on a Low-Resource General-Purpose Programming Language
por: Cheng, Junhang, et al.
Publicado: (2026)

BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
por: Zhuo, Terry Yue, et al.
Publicado: (2024)

LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks
por: Zhou, Xin, et al.
Publicado: (2025)

ProjDevBench: Benchmarking AI Coding Agents on End-to-End Project Development
por: Lu, Pengrui, et al.
Publicado: (2026)

RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation
por: Wang, Yanli, et al.
Publicado: (2024)

EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories
por: Li, Jia, et al.
Publicado: (2024)

SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios
por: Chen, Junkai, et al.
Publicado: (2025)

SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers
por: Xiang, Yanzheng, et al.
Publicado: (2025)

SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades
por: Lam, Man Ho, et al.
Publicado: (2026)

PBT-Bench: Benchmarking AI Agents on Property-Based Testing
por: Jing, Lucas, et al.
Publicado: (2026)

Towards Automated Smart Contract Generation: Evaluation, Benchmarking, and Retrieval-Augmented Repair
por: Chen, Zaoyu, et al.
Publicado: (2025)

DafnyBench: A Benchmark for Formal Software Verification
por: Loughridge, Chloe, et al.
Publicado: (2024)

Dissecting the SWE-Bench Leaderboards: Profiling Submitters and Architectures of LLM- and Agent-Based Repair Systems
por: Martinez, Matias, et al.
Publicado: (2025)

AdaptiveGuard: Towards Adaptive Runtime Safety for LLM-Powered Software
por: Yang, Rui, et al.
Publicado: (2025)

Top General Performance = Top Domain Performance? DomainCodeBench: A Multi-domain Code Generation Benchmark
por: Zheng, Dewu, et al.
Publicado: (2024)

FeatureBench: Benchmarking Agentic Coding for Complex Feature Development
por: Zhou, Qixing, et al.
Publicado: (2026)

PPM: Automated Generation of Diverse Programming Problems for Benchmarking Code Generation Models
por: Chen, Simin, et al.
Publicado: (2024)

Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
por: Merrill, Mike A., et al.
Publicado: (2026)

CSR-Bench: Benchmarking LLM Agents in Deployment of Computer Science Research Repositories
por: Xiao, Yijia, et al.
Publicado: (2025)

Drift-Bench: Diagnosing Cooperative Breakdowns in LLM Agents under Input Faults via Multi-Turn Interaction
por: Bao, Han, et al.
Publicado: (2026)

ResearchEnvBench: Benchmarking Agents on Environment Synthesis for Research Code Execution
por: Wang, Yubang, et al.
Publicado: (2026)

Benchmarking LLM Code Generation for Audio Programming with Visual Dataflow Languages
por: Zhang, William, et al.
Publicado: (2024)