TRAJECT-Bench:A Trajectory-Aware Benchmark for Evaluating Agentic Tool Use
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908587074256896 |
|---|---|
| author | He, Pengfei Dai, Zhenwei He, Bing Liu, Hui Tang, Xianfeng Lu, Hanqing Li, Juanhui Ding, Jiayuan Mukherjee, Subhabrata Wang, Suhang Xing, Yue Tang, Jiliang Dumoulin, Benoit |
| author_facet | He, Pengfei Dai, Zhenwei He, Bing Liu, Hui Tang, Xianfeng Lu, Hanqing Li, Juanhui Ding, Jiayuan Mukherjee, Subhabrata Wang, Suhang Xing, Yue Tang, Jiliang Dumoulin, Benoit |
| contents | Large language model (LLM)-based agents increasingly rely on tool use to complete real-world tasks. While existing works evaluate the LLMs' tool use capability, they largely focus on the final answers yet overlook the detailed tool usage trajectory, i.e., whether tools are selected, parameterized, and ordered correctly. We introduce TRAJECT-Bench, a trajectory-aware benchmark to comprehensively evaluate LLMs' tool use capability through diverse tasks with fine-grained evaluation metrics. TRAJECT-Bench pairs high-fidelity, executable tools across practical domains with tasks grounded in production-style APIs, and synthesizes trajectories that vary in breadth (parallel calls) and depth (interdependent chains). Besides final accuracy, TRAJECT-Bench also reports trajectory-level diagnostics, including tool selection and argument correctness, and dependency/order satisfaction. Analyses reveal failure modes such as similar tool confusion and parameter-blind selection, and scaling behavior with tool diversity and trajectory length where the bottleneck of transiting from short to mid-length trajectories is revealed, offering actionable guidance for LLMs' tool use. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_04550 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | TRAJECT-Bench:A Trajectory-Aware Benchmark for Evaluating Agentic Tool Use He, Pengfei Dai, Zhenwei He, Bing Liu, Hui Tang, Xianfeng Lu, Hanqing Li, Juanhui Ding, Jiayuan Mukherjee, Subhabrata Wang, Suhang Xing, Yue Tang, Jiliang Dumoulin, Benoit Artificial Intelligence Large language model (LLM)-based agents increasingly rely on tool use to complete real-world tasks. While existing works evaluate the LLMs' tool use capability, they largely focus on the final answers yet overlook the detailed tool usage trajectory, i.e., whether tools are selected, parameterized, and ordered correctly. We introduce TRAJECT-Bench, a trajectory-aware benchmark to comprehensively evaluate LLMs' tool use capability through diverse tasks with fine-grained evaluation metrics. TRAJECT-Bench pairs high-fidelity, executable tools across practical domains with tasks grounded in production-style APIs, and synthesizes trajectories that vary in breadth (parallel calls) and depth (interdependent chains). Besides final accuracy, TRAJECT-Bench also reports trajectory-level diagnostics, including tool selection and argument correctness, and dependency/order satisfaction. Analyses reveal failure modes such as similar tool confusion and parameter-blind selection, and scaling behavior with tool diversity and trajectory length where the bottleneck of transiting from short to mid-length trajectories is revealed, offering actionable guidance for LLMs' tool use. |
| title | TRAJECT-Bench:A Trajectory-Aware Benchmark for Evaluating Agentic Tool Use |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2510.04550 |