Firefly: Illuminating Large-Scale Verified Tool-Call Data Generation from Real APIs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914575610281984 |
|---|---|
| author | Lu, Yuxuan Wang, Ziyi Lu, Yingzhou Sang, Yisi Gesi, Jiri Tang, Xianfeng Zhang, Yimeng Dai, Zhenwei Liu, Hui Lu, Hanqing Luo, Chen He, Qi Dumoulin, Benoit Huang, Jing Wang, Dakuo |
| author_facet | Lu, Yuxuan Wang, Ziyi Lu, Yingzhou Sang, Yisi Gesi, Jiri Tang, Xianfeng Zhang, Yimeng Dai, Zhenwei Liu, Hui Lu, Hanqing Luo, Chen He, Qi Dumoulin, Benoit Huang, Jing Wang, Dakuo |
| contents | Training tool-calling agents requires large-scale trajectory data with verifiable labels, yet existing approaches either synthesize environments that diverge from real API behavior or generate tasks without ground-truth outcomes for verification. We present FireFly, a pipeline for generating verified tool-call data from real-world MCP servers. Our key insight is to invert the standard synthesis pipeline: rather than generating tasks and hoping they are solvable, we first let a strong LLM explore real APIs along graph-guided DAG structures, then synthesize tasks backward from observed outcomes, guaranteeing label correctness by construction. To handle the scale of real-world tool spaces (${\sim}$1,000 tools), we build a pairwise tool graph and sample sub-DAGs to focus exploration on semantically coherent workflows. To address environment drift in live APIs, we construct a retrieval-augmented simulator that caches all exploration results and replays them during training and evaluation, enabling fully offline and reproducible RL. Applying this pipeline yields 5,144 verified tasks spanning 240 servers and 993 tools. A 4B-parameter model trained with GRPO on FireFly matches Claude Sonnet 4.6 on our held-out test set and shows improvements on multiple tool-calling benchmarks including Tau2-Bench, MCPMark, and MCP-Atlas. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_17558 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Firefly: Illuminating Large-Scale Verified Tool-Call Data Generation from Real APIs Lu, Yuxuan Wang, Ziyi Lu, Yingzhou Sang, Yisi Gesi, Jiri Tang, Xianfeng Zhang, Yimeng Dai, Zhenwei Liu, Hui Lu, Hanqing Luo, Chen He, Qi Dumoulin, Benoit Huang, Jing Wang, Dakuo Software Engineering Computation and Language Training tool-calling agents requires large-scale trajectory data with verifiable labels, yet existing approaches either synthesize environments that diverge from real API behavior or generate tasks without ground-truth outcomes for verification. We present FireFly, a pipeline for generating verified tool-call data from real-world MCP servers. Our key insight is to invert the standard synthesis pipeline: rather than generating tasks and hoping they are solvable, we first let a strong LLM explore real APIs along graph-guided DAG structures, then synthesize tasks backward from observed outcomes, guaranteeing label correctness by construction. To handle the scale of real-world tool spaces (${\sim}$1,000 tools), we build a pairwise tool graph and sample sub-DAGs to focus exploration on semantically coherent workflows. To address environment drift in live APIs, we construct a retrieval-augmented simulator that caches all exploration results and replays them during training and evaluation, enabling fully offline and reproducible RL. Applying this pipeline yields 5,144 verified tasks spanning 240 servers and 993 tools. A 4B-parameter model trained with GRPO on FireFly matches Claude Sonnet 4.6 on our held-out test set and shows improvements on multiple tool-calling benchmarks including Tau2-Bench, MCPMark, and MCP-Atlas. |
| title | Firefly: Illuminating Large-Scale Verified Tool-Call Data Generation from Real APIs |
| topic | Software Engineering Computation and Language |
| url | https://arxiv.org/abs/2605.17558 |