Image de couverture de livre

Benchmarking Failures in Tool-Augmented Language Models

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Treviño, Eduardo, Contant, Hugo, Ngai, James, Neubig, Graham, Wang, Zora Zhiruo
Format:	Preprint
Publié:	2025
Sujets:	Software Engineering Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

CodeRAG-Bench: Can Retrieval Augment Code Generation?
par: Wang, Zora Zhiruo, et autres
Publié: (2024)

TOM-SWE: User Mental Modeling For Software Engineering Agents
par: Zhou, Xuhui, et autres
Publié: (2025)

cAST: Enhancing Code Retrieval-Augmented Generation with Structural Chunking via Abstract Syntax Tree
par: Zhang, Yilin, et autres
Publié: (2025)

Offloading Score: Measuring AI Reliance Through Counterfactual Workflows
par: Padmakumar, Vishakh, et autres
Publié: (2026)

MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use
par: Huang, Yue, et autres
Publié: (2023)

RCAgent: Cloud Root Cause Analysis by Autonomous Agents with Tool-Augmented Large Language Models
par: Wang, Zefan, et autres
Publié: (2023)

Training Versatile Coding Agents in Synthetic Environments
par: Zhu, Yiqi, et autres
Publié: (2025)

Training Software Engineering Agents and Verifiers with SWE-Gym
par: Pan, Jiayi, et autres
Publié: (2024)

Is Your Benchmark (Still) Useful? Dynamic Benchmarking for Code Language Models
par: Guan, Batu, et autres
Publié: (2025)

DI-BENCH: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scale
par: Zhang, Linghao, et autres
Publié: (2025)

Mercury: A Code Efficiency Benchmark for Code Large Language Models
par: Du, Mingzhe, et autres
Publié: (2024)

CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language Models in Tool-Calling Error Scenarios
par: Huang, Shiting, et autres
Publié: (2025)

Large Language Models are Qualified Benchmark Builders: Rebuilding Pre-Training Datasets for Advancing Code Intelligence Tasks
par: Yang, Kang, et autres
Publié: (2025)

AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators
par: Chou, Jason, et autres
Publié: (2025)

Mass-Producing Failures of Multimodal Systems with Language Models
par: Tong, Shengbang, et autres
Publié: (2023)

The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestration
par: Xu, Haoyuan, et autres
Publié: (2026)

Benchmarks as Microscopes: A Call for Model Metrology
par: Saxon, Michael, et autres
Publié: (2024)

Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?
par: Zhu, Wang Bill, et autres
Publié: (2026)

ProbeLLM: Automating Principled Diagnosis of LLM Failures
par: Huang, Yue, et autres
Publié: (2026)

ToolMATH: A Diagnostic Benchmark for Long-Horizon Tool Use under Systematic Tool-Catalog Constraints
par: Choi, Hyeonje, et autres
Publié: (2026)

ToolScope: Enhancing LLM Agent Tool Use through Tool Merging and Context-Aware Filtering
par: Liu, Marianne Menglin, et autres
Publié: (2025)

Inducing Programmatic Skills for Agentic Tasks
par: Wang, Zora Zhiruo, et autres
Publié: (2025)

Agent Workflow Memory
par: Wang, Zora Zhiruo, et autres
Publié: (2024)

R2C2-Coder: Enhancing and Benchmarking Real-world Repository-level Code Completion Abilities of Code Large Language Models
par: Deng, Ken, et autres
Publié: (2024)

Isolating Language-Coding from Problem-Solving: Benchmarking LLMs with PseudoEval
par: Wu, Jiarong, et autres
Publié: (2025)

Hybrid-Gym: Training Coding Agents to Generalize Across Tasks
par: Xie, Yiqing, et autres
Publié: (2026)

Sphinx: Benchmarking and Modeling for LLM-Driven Pull Request Review
par: Zhang, Daoan, et autres
Publié: (2026)

Demystifying and Extracting Fault-indicating Information from Logs for Failure Diagnosis
par: Huang, Junjie, et autres
Publié: (2024)

SkillCraft: Can LLM Agents Learn to Use Tools Skillfully?
par: Chen, Shiqi, et autres
Publié: (2026)

A Code Comprehension Benchmark for Large Language Models for Code
par: Havare, Jayant, et autres
Publié: (2025)

Beyond Accuracy: A Cognitive Load Framework for Mapping the Capability Boundaries of Tool-use Agents
par: Wang, Qihao, et autres
Publié: (2026)

Sanskrit Knowledge-based Systems: Annotation and Computational Tools
par: Terdalkar, Hrishikesh
Publié: (2024)

Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detection
par: Li, Yuxi, et autres
Publié: (2024)

Firefly: Illuminating Large-Scale Verified Tool-Call Data Generation from Real APIs
par: Lu, Yuxuan, et autres
Publié: (2026)

DependEval: Benchmarking LLMs for Repository Dependency Understanding
par: Du, Junjia, et autres
Publié: (2025)

Illocutionary Explanation Planning for Source-Faithful Explanations in Retrieval-Augmented Language Models
par: Sovrano, Francesco, et autres
Publié: (2026)

CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specification Generation
par: Chen, Zaoyu, et autres
Publié: (2026)

From Proof to Program: Characterizing Tool-Induced Reasoning Hallucinations in Large Language Models
par: Bayat, Farima Fatahi, et autres
Publié: (2025)

Breaking the Cycle of Recurring Failures: Applying Generative AI to Root Cause Analysis in Legacy Banking Systems
par: Jin, Siyuan, et autres
Publié: (2024)

TimeMachine-bench: A Benchmark for Evaluating Model Capabilities in Repository-Level Migration Tasks
par: Fujii, Ryo, et autres
Publié: (2026)