Image de couverture de livre

BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Diddee, Harshita, Yauney, Gregory, Swayamdipta, Swabha, Ippolito, Daphne
Format:	Preprint
Publié:	2026
Sujets:	Computation and Language Artificial Intelligence Software Engineering
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Chasing Random: Instruction Selection Strategies Fail to Generalize
par: Diddee, Harshita, et autres
Publié: (2024)

CIE: Controlling Language Model Text Generations Using Continuous Signals
par: Samuel, Vinay, et autres
Publié: (2025)

UA-Code-Bench: A Competitive Programming Benchmark for Evaluating LLM Code Generation in Ukrainian
par: Syromiatnikov, Mykyta, et autres
Publié: (2025)

BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks
par: Tu, Xinming, et autres
Publié: (2026)

Towards Automated Smart Contract Generation: Evaluation, Benchmarking, and Retrieval-Augmented Repair
par: Chen, Zaoyu, et autres
Publié: (2025)

How Reliable is Language Model Micro-Benchmarking?
par: Yauney, Gregory, et autres
Publié: (2025)

ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development
par: Yang, Jie, et autres
Publié: (2026)

CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks
par: Jiang, Hongchao, et autres
Publié: (2025)

Collu-Bench: A Benchmark for Predicting Language Model Hallucinations in Code
par: Jiang, Nan, et autres
Publié: (2024)

CangjieBench: Benchmarking LLMs on a Low-Resource General-Purpose Programming Language
par: Cheng, Junhang, et autres
Publié: (2026)

BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
par: Zhuo, Terry Yue, et autres
Publié: (2024)

DebugBench: Evaluating Debugging Capability of Large Language Models
par: Tian, Runchu, et autres
Publié: (2024)

RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation
par: Wang, Yanli, et autres
Publié: (2024)

EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories
par: Li, Jia, et autres
Publié: (2024)

GameDevBench: Evaluating Agentic Capabilities Through Game Development
par: Chi, Wayne, et autres
Publié: (2026)

SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks
par: Orlanski, Gabriel, et autres
Publié: (2026)

CR-Bench: Evaluating the Real-World Utility of AI Code Review Agents
par: Pereira, Kristen, et autres
Publié: (2026)

FeatBench: Towards More Realistic Evaluation of Feature-level Code Generation
par: Chen, Haorui, et autres
Publié: (2025)

LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks
par: Zhou, Xin, et autres
Publié: (2025)

Top General Performance = Top Domain Performance? DomainCodeBench: A Multi-domain Code Generation Benchmark
par: Zheng, Dewu, et autres
Publié: (2024)

Customizing Large Language Model Generation Style using Parameter-Efficient Finetuning
par: Liu, Xinyue, et autres
Publié: (2024)

OmniCode: A Benchmark for Evaluating Software Engineering Agents
par: Sonwane, Atharv, et autres
Publié: (2026)

Benchmarking Open-Source Safety Guard Models: A Comprehensive Evaluation
par: Harsh, Reetu Raj, et autres
Publié: (2026)

Evaluating Retrieval-Augmented Generation Variants for Natural Language-Based SQL and API Call Generation
par: Marketsmüller, Michael, et autres
Publié: (2026)

SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents
par: Zhao, Bingchen, et autres
Publié: (2026)

SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks
par: Adamenko, Pavel, et autres
Publié: (2025)

CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation
par: Yan, Weixiang, et autres
Publié: (2023)

AInsteinBench: Benchmarking Coding Agents on Scientific Repositories
par: Duston, Titouan, et autres
Publié: (2025)

Dissecting the SWE-Bench Leaderboards: Profiling Submitters and Architectures of LLM- and Agent-Based Repair Systems
par: Martinez, Matias, et autres
Publié: (2025)

MacroBench: A Novel Testbed for Web Automation Scripts via Large Language Models
par: Kim, Hyunjun, et autres
Publié: (2025)

SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers
par: Xiang, Yanzheng, et autres
Publié: (2025)

Drift-Bench: Diagnosing Cooperative Breakdowns in LLM Agents under Input Faults via Multi-Turn Interaction
par: Bao, Han, et autres
Publié: (2026)

SWE-Bench++: A Framework for the Scalable Generation of Software Engineering Benchmarks from Open-Source Repositories
par: Wang, Lilin, et autres
Publié: (2025)

CodeEditorBench: Evaluating Code Editing Capability of Large Language Models
par: Guo, Jiawei, et autres
Publié: (2024)

PERC: Plan-As-Query Example Retrieval for Underrepresented Code Generation
par: Yoo, Jaeseok, et autres
Publié: (2024)

Using Grammar Masking to Ensure Syntactic Validity in LLM-based Modeling Tasks
par: Netz, Lukas, et autres
Publié: (2024)

SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios
par: Chen, Junkai, et autres
Publié: (2025)

IndustryCode: A Benchmark for Industry Code Generation
par: Zeng, Puyu, et autres
Publié: (2026)

Illocutionary Explanation Planning for Source-Faithful Explanations in Retrieval-Augmented Language Models
par: Sovrano, Francesco, et autres
Publié: (2026)

BuildBench: Benchmarking LLM Agents on Compiling Real-World Open-Source Software
par: Zhang, Zehua, et autres
Publié: (2025)