Benchmarking Failures in Tool-Augmented Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | Treviño, Eduardo, Contant, Hugo, Ngai, James, Neubig, Graham, Wang, Zora Zhiruo |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
Ejemplares similares
CodeRAG-Bench: Can Retrieval Augment Code Generation?
por: Wang, Zora Zhiruo, et al.
Publicado: (2024)
por: Wang, Zora Zhiruo, et al.
Publicado: (2024)
TOM-SWE: User Mental Modeling For Software Engineering Agents
por: Zhou, Xuhui, et al.
Publicado: (2025)
por: Zhou, Xuhui, et al.
Publicado: (2025)
cAST: Enhancing Code Retrieval-Augmented Generation with Structural Chunking via Abstract Syntax Tree
por: Zhang, Yilin, et al.
Publicado: (2025)
por: Zhang, Yilin, et al.
Publicado: (2025)
Offloading Score: Measuring AI Reliance Through Counterfactual Workflows
por: Padmakumar, Vishakh, et al.
Publicado: (2026)
por: Padmakumar, Vishakh, et al.
Publicado: (2026)
MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use
por: Huang, Yue, et al.
Publicado: (2023)
por: Huang, Yue, et al.
Publicado: (2023)
RCAgent: Cloud Root Cause Analysis by Autonomous Agents with Tool-Augmented Large Language Models
por: Wang, Zefan, et al.
Publicado: (2023)
por: Wang, Zefan, et al.
Publicado: (2023)
Training Versatile Coding Agents in Synthetic Environments
por: Zhu, Yiqi, et al.
Publicado: (2025)
por: Zhu, Yiqi, et al.
Publicado: (2025)
Training Software Engineering Agents and Verifiers with SWE-Gym
por: Pan, Jiayi, et al.
Publicado: (2024)
por: Pan, Jiayi, et al.
Publicado: (2024)
Is Your Benchmark (Still) Useful? Dynamic Benchmarking for Code Language Models
por: Guan, Batu, et al.
Publicado: (2025)
por: Guan, Batu, et al.
Publicado: (2025)
DI-BENCH: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scale
por: Zhang, Linghao, et al.
Publicado: (2025)
por: Zhang, Linghao, et al.
Publicado: (2025)
Mercury: A Code Efficiency Benchmark for Code Large Language Models
por: Du, Mingzhe, et al.
Publicado: (2024)
por: Du, Mingzhe, et al.
Publicado: (2024)
CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language Models in Tool-Calling Error Scenarios
por: Huang, Shiting, et al.
Publicado: (2025)
por: Huang, Shiting, et al.
Publicado: (2025)
Large Language Models are Qualified Benchmark Builders: Rebuilding Pre-Training Datasets for Advancing Code Intelligence Tasks
por: Yang, Kang, et al.
Publicado: (2025)
por: Yang, Kang, et al.
Publicado: (2025)
AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators
por: Chou, Jason, et al.
Publicado: (2025)
por: Chou, Jason, et al.
Publicado: (2025)
Mass-Producing Failures of Multimodal Systems with Language Models
por: Tong, Shengbang, et al.
Publicado: (2023)
por: Tong, Shengbang, et al.
Publicado: (2023)
The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestration
por: Xu, Haoyuan, et al.
Publicado: (2026)
por: Xu, Haoyuan, et al.
Publicado: (2026)
Benchmarks as Microscopes: A Call for Model Metrology
por: Saxon, Michael, et al.
Publicado: (2024)
por: Saxon, Michael, et al.
Publicado: (2024)
Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?
por: Zhu, Wang Bill, et al.
Publicado: (2026)
por: Zhu, Wang Bill, et al.
Publicado: (2026)
ProbeLLM: Automating Principled Diagnosis of LLM Failures
por: Huang, Yue, et al.
Publicado: (2026)
por: Huang, Yue, et al.
Publicado: (2026)
ToolMATH: A Diagnostic Benchmark for Long-Horizon Tool Use under Systematic Tool-Catalog Constraints
por: Choi, Hyeonje, et al.
Publicado: (2026)
por: Choi, Hyeonje, et al.
Publicado: (2026)
ToolScope: Enhancing LLM Agent Tool Use through Tool Merging and Context-Aware Filtering
por: Liu, Marianne Menglin, et al.
Publicado: (2025)
por: Liu, Marianne Menglin, et al.
Publicado: (2025)
Inducing Programmatic Skills for Agentic Tasks
por: Wang, Zora Zhiruo, et al.
Publicado: (2025)
por: Wang, Zora Zhiruo, et al.
Publicado: (2025)
Agent Workflow Memory
por: Wang, Zora Zhiruo, et al.
Publicado: (2024)
por: Wang, Zora Zhiruo, et al.
Publicado: (2024)
R2C2-Coder: Enhancing and Benchmarking Real-world Repository-level Code Completion Abilities of Code Large Language Models
por: Deng, Ken, et al.
Publicado: (2024)
por: Deng, Ken, et al.
Publicado: (2024)
Isolating Language-Coding from Problem-Solving: Benchmarking LLMs with PseudoEval
por: Wu, Jiarong, et al.
Publicado: (2025)
por: Wu, Jiarong, et al.
Publicado: (2025)
Hybrid-Gym: Training Coding Agents to Generalize Across Tasks
por: Xie, Yiqing, et al.
Publicado: (2026)
por: Xie, Yiqing, et al.
Publicado: (2026)
Sphinx: Benchmarking and Modeling for LLM-Driven Pull Request Review
por: Zhang, Daoan, et al.
Publicado: (2026)
por: Zhang, Daoan, et al.
Publicado: (2026)
Demystifying and Extracting Fault-indicating Information from Logs for Failure Diagnosis
por: Huang, Junjie, et al.
Publicado: (2024)
por: Huang, Junjie, et al.
Publicado: (2024)
SkillCraft: Can LLM Agents Learn to Use Tools Skillfully?
por: Chen, Shiqi, et al.
Publicado: (2026)
por: Chen, Shiqi, et al.
Publicado: (2026)
A Code Comprehension Benchmark for Large Language Models for Code
por: Havare, Jayant, et al.
Publicado: (2025)
por: Havare, Jayant, et al.
Publicado: (2025)
Beyond Accuracy: A Cognitive Load Framework for Mapping the Capability Boundaries of Tool-use Agents
por: Wang, Qihao, et al.
Publicado: (2026)
por: Wang, Qihao, et al.
Publicado: (2026)
Sanskrit Knowledge-based Systems: Annotation and Computational Tools
por: Terdalkar, Hrishikesh
Publicado: (2024)
por: Terdalkar, Hrishikesh
Publicado: (2024)
Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detection
por: Li, Yuxi, et al.
Publicado: (2024)
por: Li, Yuxi, et al.
Publicado: (2024)
Firefly: Illuminating Large-Scale Verified Tool-Call Data Generation from Real APIs
por: Lu, Yuxuan, et al.
Publicado: (2026)
por: Lu, Yuxuan, et al.
Publicado: (2026)
DependEval: Benchmarking LLMs for Repository Dependency Understanding
por: Du, Junjia, et al.
Publicado: (2025)
por: Du, Junjia, et al.
Publicado: (2025)
Illocutionary Explanation Planning for Source-Faithful Explanations in Retrieval-Augmented Language Models
por: Sovrano, Francesco, et al.
Publicado: (2026)
por: Sovrano, Francesco, et al.
Publicado: (2026)
CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specification Generation
por: Chen, Zaoyu, et al.
Publicado: (2026)
por: Chen, Zaoyu, et al.
Publicado: (2026)
From Proof to Program: Characterizing Tool-Induced Reasoning Hallucinations in Large Language Models
por: Bayat, Farima Fatahi, et al.
Publicado: (2025)
por: Bayat, Farima Fatahi, et al.
Publicado: (2025)
Breaking the Cycle of Recurring Failures: Applying Generative AI to Root Cause Analysis in Legacy Banking Systems
por: Jin, Siyuan, et al.
Publicado: (2024)
por: Jin, Siyuan, et al.
Publicado: (2024)
TimeMachine-bench: A Benchmark for Evaluating Model Capabilities in Repository-Level Migration Tasks
por: Fujii, Ryo, et al.
Publicado: (2026)
por: Fujii, Ryo, et al.
Publicado: (2026)
Ejemplares similares
-
CodeRAG-Bench: Can Retrieval Augment Code Generation?
por: Wang, Zora Zhiruo, et al.
Publicado: (2024) -
TOM-SWE: User Mental Modeling For Software Engineering Agents
por: Zhou, Xuhui, et al.
Publicado: (2025) -
cAST: Enhancing Code Retrieval-Augmented Generation with Structural Chunking via Abstract Syntax Tree
por: Zhang, Yilin, et al.
Publicado: (2025) -
Offloading Score: Measuring AI Reliance Through Counterfactual Workflows
por: Padmakumar, Vishakh, et al.
Publicado: (2026) -
MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use
por: Huang, Yue, et al.
Publicado: (2023)