Cover Image

SynAE: A Framework for Measuring the Quality of Synthetic Data for Tool-Calling Agent Evaluations

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Wang, Shuaiqi, Maddi, Aadyaa, Lin, Zinan, Fanti, Giulia
Format:	Preprint
Published:	2026
Subjects:	Computation and Language Machine Learning Software Engineering
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Quality Matters: Evaluating Synthetic Data for Tool-Using LLMs
by: Iskander, Shadi, et al.
Published: (2024)

Tool Calling is Linearly Readable and Steerable in Language Models
by: Wu, Zekun, et al.
Published: (2026)

Code Review Without Borders: Evaluating Synthetic vs. Real Data for Review Recommendation
by: Cohen, Yogev, et al.
Published: (2025)

ToolRegistry: A Protocol-Agnostic Tool Management Library for Function-Calling LLMs
by: Ding, Peng, et al.
Published: (2025)

SynthTools: A Framework for Scaling Synthetic Tools for Agent Development
by: Castellani, Tommaso, et al.
Published: (2025)

An Exploratory Study on Automatic Identification of Assumptions in the Development of Deep Learning Frameworks
by: Yang, Chen, et al.
Published: (2024)

SynCode: LLM Generation with Grammar Augmentation
by: Ugare, Shubham, et al.
Published: (2024)

Learning Code Preference via Synthetic Evolution
by: Liu, Jiawei, et al.
Published: (2024)

ToolMATH: A Diagnostic Benchmark for Long-Horizon Tool Use under Systematic Tool-Catalog Constraints
by: Choi, Hyeonje, et al.
Published: (2026)

CodeVisionary: An Agent-based Framework for Evaluating Large Language Models in Code Generation
by: Wang, Xinchen, et al.
Published: (2025)

TraceSafe: A Systematic Assessment of LLM Guardrails on Multi-Step Tool-Calling Trajectories
by: Chen, Yen-Shan, et al.
Published: (2026)

SWE-Debate: Competitive Multi-Agent Debate for Software Issue Resolution
by: Li, Han, et al.
Published: (2025)

The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestration
by: Xu, Haoyuan, et al.
Published: (2026)

GrowthHacker: Automated Off-Policy Evaluation Optimization Using Code-Modifying LLM Agents
by: Wu, Jie JW, et al.
Published: (2025)

Scoring Verifiers: Evaluating Synthetic Verification for Code and Reasoning
by: Ficek, Aleksander, et al.
Published: (2025)

SEVerA: Verified Synthesis of Self-Evolving Agents
by: Banerjee, Debangshu, et al.
Published: (2026)

Understanding Tool-Augmented Agents for Lean Formalization: A Factorial Analysis
by: Zhang, Ke, et al.
Published: (2026)

SceneGenAgent: Precise Industrial Scene Generation with Coding Agent
by: Xia, Xiao, et al.
Published: (2024)

Measuring memorization in RLHF for code completion
by: Pappu, Aneesh, et al.
Published: (2024)

Hybrid-Gym: Training Coding Agents to Generalize Across Tasks
by: Xie, Yiqing, et al.
Published: (2026)

Evaluating Language Models for Efficient Code Generation
by: Liu, Jiawei, et al.
Published: (2024)

APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets
by: Liu, Zuxin, et al.
Published: (2024)

CIDRe: A Reference-Free Multi-Aspect Criterion for Code Comment Quality Measurement
by: Dziuba, Maria, et al.
Published: (2025)

ToolFactory: Automating Tool Generation by Leveraging LLM to Understand REST API Documentations
by: Ni, Xinyi, et al.
Published: (2025)

SERA: Soft-Verified Efficient Repository Agents
by: Shen, Ethan, et al.
Published: (2026)

RepoQA: Evaluating Long Context Code Understanding
by: Liu, Jiawei, et al.
Published: (2024)

Exploring LLM-based Agents for Root Cause Analysis
by: Roy, Devjeet, et al.
Published: (2024)

How Do Your Code LLMs Perform? Empowering Code Instruction Tuning with High-Quality Data
by: Wang, Yejie, et al.
Published: (2024)

On the Effectiveness of Machine Learning-based Call Graph Pruning: An Empirical Study
by: Mir, Amir M., et al.
Published: (2024)

CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language Models in Tool-Calling Error Scenarios
by: Huang, Shiting, et al.
Published: (2025)

LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs
by: Xia, Yunhui, et al.
Published: (2025)

Training Long-Context, Multi-Turn Software Engineering Agents with Reinforcement Learning
by: Golubev, Alexander, et al.
Published: (2025)

LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
by: Jain, Naman, et al.
Published: (2024)

GSO: Challenging Software Optimization Tasks for Evaluating SWE-Agents
by: Shetty, Manish, et al.
Published: (2025)

FROAV: A Framework for RAG Observation and Agent Verification -- Lowering the Barrier to LLM Agent Research
by: Lin, Tzu-Hsuan, et al.
Published: (2026)

Confucius Code Agent: Scalable Agent Scaffolding for Real-World Codebases
by: Wong, Sherman, et al.
Published: (2025)

Language Models for Code Completion: A Practical Evaluation
by: Izadi, Maliheh, et al.
Published: (2024)

Evaluation and Improvement of Fault Detection for Large Language Models
by: Hu, Qiang, et al.
Published: (2024)

CodeTool: Enhancing Programmatic Tool Invocation of LLMs via Process Supervision
by: Lu, Yifei, et al.
Published: (2025)

CodeJudge: Evaluating Code Generation with Large Language Models
by: Tong, Weixi, et al.
Published: (2024)