Training LLMs for Multi-Step Tool Orchestration with Constrained Data Synthesis and Graduated Rewards
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910107051229184 |
|---|---|
| author | Jiayang, Cheng Liu, Xin Zhang, Zhihan Wen, Haoyang Zhang, Zixuan Yin, Qingyu Li, Shiyang Nigam, Priyanka Yin, Bing Zhang, Chao Song, Yangqiu |
| author_facet | Jiayang, Cheng Liu, Xin Zhang, Zhihan Wen, Haoyang Zhang, Zixuan Yin, Qingyu Li, Shiyang Nigam, Priyanka Yin, Bing Zhang, Chao Song, Yangqiu |
| contents | Multi-step tool orchestration remains challenging for LLMs, as state-of-the-art models frequently fail on full sequence execution due to parameter errors. Training for these workflows faces two obstacles: the lack of environments supporting complex real-world API dependencies, and sparse binary rewards that provide no signal for partial correctness. We propose a reinforcement learning framework addressing both challenges. First, we construct a deterministic environment backed by a large-scale cache of real API responses, enabling constrained synthesis of valid multi-step traces with controllable complexity. Second, we introduce a graduated reward that decomposes correctness into atomic validity (call-level correctness at increasing granularity) and orchestration consistency (correct sequencing with dependency respect). On ComplexFuncBench, our approach substantially improves turn accuracy, with ablations confirming both reward components are essential. Cross-benchmark evaluation on BFCL v4 shows that the learned orchestration skills transfer to entirely different API ecosystems (e.g., agentic web search and memory management), yielding consistent gains while maintaining stable single-step performance. Code is available at https://github.com/horizon-rl/ToolOrchestrationReward |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_24709 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Training LLMs for Multi-Step Tool Orchestration with Constrained Data Synthesis and Graduated Rewards Jiayang, Cheng Liu, Xin Zhang, Zhihan Wen, Haoyang Zhang, Zixuan Yin, Qingyu Li, Shiyang Nigam, Priyanka Yin, Bing Zhang, Chao Song, Yangqiu Machine Learning Computation and Language Multi-step tool orchestration remains challenging for LLMs, as state-of-the-art models frequently fail on full sequence execution due to parameter errors. Training for these workflows faces two obstacles: the lack of environments supporting complex real-world API dependencies, and sparse binary rewards that provide no signal for partial correctness. We propose a reinforcement learning framework addressing both challenges. First, we construct a deterministic environment backed by a large-scale cache of real API responses, enabling constrained synthesis of valid multi-step traces with controllable complexity. Second, we introduce a graduated reward that decomposes correctness into atomic validity (call-level correctness at increasing granularity) and orchestration consistency (correct sequencing with dependency respect). On ComplexFuncBench, our approach substantially improves turn accuracy, with ablations confirming both reward components are essential. Cross-benchmark evaluation on BFCL v4 shows that the learned orchestration skills transfer to entirely different API ecosystems (e.g., agentic web search and memory management), yielding consistent gains while maintaining stable single-step performance. Code is available at https://github.com/horizon-rl/ToolOrchestrationReward |
| title | Training LLMs for Multi-Step Tool Orchestration with Constrained Data Synthesis and Graduated Rewards |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2603.24709 |