ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918026154082304 |
|---|---|
| author | Ye, Junjie Du, Zhengyin Yao, Xuesong Lin, Weijian Xu, Yufei Chen, Zehui Wang, Zaiyuan Zhu, Sining Xi, Zhiheng Yuan, Siyu Gui, Tao Zhang, Qi Huang, Xuanjing Chen, Jiecao |
| author_facet | Ye, Junjie Du, Zhengyin Yao, Xuesong Lin, Weijian Xu, Yufei Chen, Zehui Wang, Zaiyuan Zhu, Sining Xi, Zhiheng Yuan, Siyu Gui, Tao Zhang, Qi Huang, Xuanjing Chen, Jiecao |
| contents | Effective evaluation of multi-hop tool use is critical for analyzing the understanding, reasoning, and function-calling capabilities of large language models (LLMs). However, progress has been hindered by a lack of reliable evaluation datasets. To address this, we present ToolHop, a dataset comprising 995 user queries and 3,912 associated tools, specifically designed for rigorous evaluation of multi-hop tool use. ToolHop ensures diverse queries, meaningful interdependencies, locally executable tools, detailed feedback, and verifiable answers through a novel query-driven data construction approach that includes tool creation, document refinement, and code generation. We evaluate 14 LLMs across five model families (i.e., LLaMA3.1, Qwen2.5, Gemini1.5, Claude3.5, and GPT), uncovering significant challenges in handling multi-hop tool-use scenarios. The leading model, GPT-4o, achieves an accuracy of 49.04%, underscoring substantial room for improvement. Further analysis reveals variations in tool-use strategies for various families, offering actionable insights to guide the development of more effective approaches. Code and data can be found in https://huggingface.co/datasets/bytedance-research/ToolHop. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_02506 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use Ye, Junjie Du, Zhengyin Yao, Xuesong Lin, Weijian Xu, Yufei Chen, Zehui Wang, Zaiyuan Zhu, Sining Xi, Zhiheng Yuan, Siyu Gui, Tao Zhang, Qi Huang, Xuanjing Chen, Jiecao Computation and Language Effective evaluation of multi-hop tool use is critical for analyzing the understanding, reasoning, and function-calling capabilities of large language models (LLMs). However, progress has been hindered by a lack of reliable evaluation datasets. To address this, we present ToolHop, a dataset comprising 995 user queries and 3,912 associated tools, specifically designed for rigorous evaluation of multi-hop tool use. ToolHop ensures diverse queries, meaningful interdependencies, locally executable tools, detailed feedback, and verifiable answers through a novel query-driven data construction approach that includes tool creation, document refinement, and code generation. We evaluate 14 LLMs across five model families (i.e., LLaMA3.1, Qwen2.5, Gemini1.5, Claude3.5, and GPT), uncovering significant challenges in handling multi-hop tool-use scenarios. The leading model, GPT-4o, achieves an accuracy of 49.04%, underscoring substantial room for improvement. Further analysis reveals variations in tool-use strategies for various families, offering actionable insights to guide the development of more effective approaches. Code and data can be found in https://huggingface.co/datasets/bytedance-research/ToolHop. |
| title | ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2501.02506 |