Imagen de Portada

Benchmarking Failures in Tool-Augmented Language Models

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Treviño, Eduardo, Contant, Hugo, Ngai, James, Neubig, Graham, Wang, Zora Zhiruo
Formato:	Preprint
Publicado:	2025
Materias:	Software Engineering Computation and Language
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

CodeRAG-Bench: Can Retrieval Augment Code Generation?
por: Wang, Zora Zhiruo, et al.
Publicado: (2024)

TOM-SWE: User Mental Modeling For Software Engineering Agents
por: Zhou, Xuhui, et al.
Publicado: (2025)

cAST: Enhancing Code Retrieval-Augmented Generation with Structural Chunking via Abstract Syntax Tree
por: Zhang, Yilin, et al.
Publicado: (2025)

Offloading Score: Measuring AI Reliance Through Counterfactual Workflows
por: Padmakumar, Vishakh, et al.
Publicado: (2026)

MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use
por: Huang, Yue, et al.
Publicado: (2023)

RCAgent: Cloud Root Cause Analysis by Autonomous Agents with Tool-Augmented Large Language Models
por: Wang, Zefan, et al.
Publicado: (2023)

Training Versatile Coding Agents in Synthetic Environments
por: Zhu, Yiqi, et al.
Publicado: (2025)

Training Software Engineering Agents and Verifiers with SWE-Gym
por: Pan, Jiayi, et al.
Publicado: (2024)

Is Your Benchmark (Still) Useful? Dynamic Benchmarking for Code Language Models
por: Guan, Batu, et al.
Publicado: (2025)

DI-BENCH: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scale
por: Zhang, Linghao, et al.
Publicado: (2025)

Mercury: A Code Efficiency Benchmark for Code Large Language Models
por: Du, Mingzhe, et al.
Publicado: (2024)

CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language Models in Tool-Calling Error Scenarios
por: Huang, Shiting, et al.
Publicado: (2025)

Large Language Models are Qualified Benchmark Builders: Rebuilding Pre-Training Datasets for Advancing Code Intelligence Tasks
por: Yang, Kang, et al.
Publicado: (2025)

AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators
por: Chou, Jason, et al.
Publicado: (2025)

Mass-Producing Failures of Multimodal Systems with Language Models
por: Tong, Shengbang, et al.
Publicado: (2023)

The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestration
por: Xu, Haoyuan, et al.
Publicado: (2026)

Benchmarks as Microscopes: A Call for Model Metrology
por: Saxon, Michael, et al.
Publicado: (2024)

Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?
por: Zhu, Wang Bill, et al.
Publicado: (2026)

ProbeLLM: Automating Principled Diagnosis of LLM Failures
por: Huang, Yue, et al.
Publicado: (2026)

ToolMATH: A Diagnostic Benchmark for Long-Horizon Tool Use under Systematic Tool-Catalog Constraints
por: Choi, Hyeonje, et al.
Publicado: (2026)

ToolScope: Enhancing LLM Agent Tool Use through Tool Merging and Context-Aware Filtering
por: Liu, Marianne Menglin, et al.
Publicado: (2025)

Inducing Programmatic Skills for Agentic Tasks
por: Wang, Zora Zhiruo, et al.
Publicado: (2025)

Agent Workflow Memory
por: Wang, Zora Zhiruo, et al.
Publicado: (2024)

R2C2-Coder: Enhancing and Benchmarking Real-world Repository-level Code Completion Abilities of Code Large Language Models
por: Deng, Ken, et al.
Publicado: (2024)

Isolating Language-Coding from Problem-Solving: Benchmarking LLMs with PseudoEval
por: Wu, Jiarong, et al.
Publicado: (2025)

Hybrid-Gym: Training Coding Agents to Generalize Across Tasks
por: Xie, Yiqing, et al.
Publicado: (2026)

Sphinx: Benchmarking and Modeling for LLM-Driven Pull Request Review
por: Zhang, Daoan, et al.
Publicado: (2026)

Demystifying and Extracting Fault-indicating Information from Logs for Failure Diagnosis
por: Huang, Junjie, et al.
Publicado: (2024)

SkillCraft: Can LLM Agents Learn to Use Tools Skillfully?
por: Chen, Shiqi, et al.
Publicado: (2026)

A Code Comprehension Benchmark for Large Language Models for Code
por: Havare, Jayant, et al.
Publicado: (2025)

Beyond Accuracy: A Cognitive Load Framework for Mapping the Capability Boundaries of Tool-use Agents
por: Wang, Qihao, et al.
Publicado: (2026)

Sanskrit Knowledge-based Systems: Annotation and Computational Tools
por: Terdalkar, Hrishikesh
Publicado: (2024)

Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detection
por: Li, Yuxi, et al.
Publicado: (2024)

Firefly: Illuminating Large-Scale Verified Tool-Call Data Generation from Real APIs
por: Lu, Yuxuan, et al.
Publicado: (2026)

DependEval: Benchmarking LLMs for Repository Dependency Understanding
por: Du, Junjia, et al.
Publicado: (2025)

Illocutionary Explanation Planning for Source-Faithful Explanations in Retrieval-Augmented Language Models
por: Sovrano, Francesco, et al.
Publicado: (2026)

CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specification Generation
por: Chen, Zaoyu, et al.
Publicado: (2026)

From Proof to Program: Characterizing Tool-Induced Reasoning Hallucinations in Large Language Models
por: Bayat, Farima Fatahi, et al.
Publicado: (2025)

Breaking the Cycle of Recurring Failures: Applying Generative AI to Root Cause Analysis in Legacy Banking Systems
por: Jin, Siyuan, et al.
Publicado: (2024)

TimeMachine-bench: A Benchmark for Evaluating Model Capabilities in Repository-Level Migration Tasks
por: Fujii, Ryo, et al.
Publicado: (2026)