Copertina

Benchmarks as Microscopes: A Call for Model Metrology

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Saxon, Michael, Holtzman, Ari, West, Peter, Wang, William Yang, Saphra, Naomi
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Software Engineering Computation and Language
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

CallNavi, A Challenge and Empirical Study on LLM Function Calling and Routing
di: Song, Yewei, et al.
Pubblicazione: (2025)

CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language Models in Tool-Calling Error Scenarios
di: Huang, Shiting, et al.
Pubblicazione: (2025)

Benchmarking Failures in Tool-Augmented Language Models
di: Treviño, Eduardo, et al.
Pubblicazione: (2025)

Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?
di: Zhu, Wang Bill, et al.
Pubblicazione: (2026)

BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
di: Zhuo, Terry Yue, et al.
Pubblicazione: (2024)

Firefly: Illuminating Large-Scale Verified Tool-Call Data Generation from Real APIs
di: Lu, Yuxuan, et al.
Pubblicazione: (2026)

Is Your Benchmark (Still) Useful? Dynamic Benchmarking for Code Language Models
di: Guan, Batu, et al.
Pubblicazione: (2025)

The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestration
di: Xu, Haoyuan, et al.
Pubblicazione: (2026)

Large Language Models are Qualified Benchmark Builders: Rebuilding Pre-Training Datasets for Advancing Code Intelligence Tasks
di: Yang, Kang, et al.
Pubblicazione: (2025)

Mercury: A Code Efficiency Benchmark for Code Large Language Models
di: Du, Mingzhe, et al.
Pubblicazione: (2024)

DI-BENCH: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scale
di: Zhang, Linghao, et al.
Pubblicazione: (2025)

Sphinx: Benchmarking and Modeling for LLM-Driven Pull Request Review
di: Zhang, Daoan, et al.
Pubblicazione: (2026)

TimeMachine-bench: A Benchmark for Evaluating Model Capabilities in Repository-Level Migration Tasks
di: Fujii, Ryo, et al.
Pubblicazione: (2026)

CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation
di: Wang, Sizhe, et al.
Pubblicazione: (2025)

Tool Calling is Linearly Readable and Steerable in Language Models
di: Wu, Zekun, et al.
Pubblicazione: (2026)

AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators
di: Chou, Jason, et al.
Pubblicazione: (2025)

ProjectEval: A Benchmark for Programming Agents Automated Evaluation on Project-Level Code Generation
di: Liu, Kaiyuan, et al.
Pubblicazione: (2025)

ClassEval-Pro: A Cross-Domain Benchmark for Class-Level Code Generation
di: Chen, Yeheng, et al.
Pubblicazione: (2026)

DependEval: Benchmarking LLMs for Repository Dependency Understanding
di: Du, Junjia, et al.
Pubblicazione: (2025)

CommitBench: A Benchmark for Commit Message Generation
di: Schall, Maximilian, et al.
Pubblicazione: (2024)

FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation
di: Li, Wei, et al.
Pubblicazione: (2025)

Evaluating LLMs on Sequential API Call Through Automated Test Generation
di: Huang, Yuheng, et al.
Pubblicazione: (2025)

Evaluating Retrieval-Augmented Generation Variants for Natural Language-Based SQL and API Call Generation
di: Marketsmüller, Michael, et al.
Pubblicazione: (2026)

CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specification Generation
di: Chen, Zaoyu, et al.
Pubblicazione: (2026)

MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use
di: Huang, Yue, et al.
Pubblicazione: (2023)

Converted, Not Equivalent: Benchmarking Codebase Conversion via Observational Equivalence
di: Song, Linxin, et al.
Pubblicazione: (2026)

SynAE: A Framework for Measuring the Quality of Synthetic Data for Tool-Calling Agent Evaluations
di: Wang, Shuaiqi, et al.
Pubblicazione: (2026)

DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories
di: Li, Jia, et al.
Pubblicazione: (2024)

R2C2-Coder: Enhancing and Benchmarking Real-world Repository-level Code Completion Abilities of Code Large Language Models
di: Deng, Ken, et al.
Pubblicazione: (2024)

EigenData: A Self-Evolving Multi-Agent Platform for Function-Calling Data Synthesis, Auditing, and Repair
di: Chen, Jiaao, et al.
Pubblicazione: (2026)

Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks
di: Zhao, Songwen, et al.
Pubblicazione: (2025)

ContextEcho: A Benchmark for Persona Drift in Long Agentic-Coding Sessions
di: Ding, Xianzhong, et al.
Pubblicazione: (2026)

MultiFileTest: A Multi-File-Level LLM Unit Test Generation Benchmark and Impact of Error Fixing Mechanisms
di: Wang, Yibo, et al.
Pubblicazione: (2025)

The Flaw of Averages: Quantifying Uniformity of Performance on Benchmarks
di: Uzunoglu, Arda, et al.
Pubblicazione: (2025)

A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic System
di: Guo, Jiale, et al.
Pubblicazione: (2025)

EffiBench: Benchmarking the Efficiency of Automatically Generated Code
di: Huang, Dong, et al.
Pubblicazione: (2024)

ISD-Agent-Bench: A Comprehensive Benchmark for Evaluating LLM-based Instructional Design Agents
di: Jeon, YoungHoon, et al.
Pubblicazione: (2026)

Adaptable and Precise: Enterprise-Scenario LLM Function-Calling Capability Training Pipeline
di: Zeng, Guancheng, et al.
Pubblicazione: (2024)

Testing the Effect of Code Documentation on Large Language Model Code Understanding
di: Macke, William, et al.
Pubblicazione: (2024)

CodeUpdateArena: Benchmarking Knowledge Editing on API Updates
di: Liu, Zeyu Leo, et al.
Pubblicazione: (2024)