NesTools: A Dataset for Evaluating Nested Tool Learning Abilities of Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912179064668160 |
|---|---|
| author | Han, Han Zhu, Tong Zhang, Xiang Wu, Mengsong Xiong, Hao Chen, Wenliang |
| author_facet | Han, Han Zhu, Tong Zhang, Xiang Wu, Mengsong Xiong, Hao Chen, Wenliang |
| contents | Large language models (LLMs) combined with tool learning have gained impressive results in real-world applications. During tool learning, LLMs may call multiple tools in nested orders, where the latter tool call may take the former response as its input parameters. However, current research on the nested tool learning capabilities is still under-explored, since the existing benchmarks lack relevant data instances. To address this problem, we introduce NesTools to bridge the current gap in comprehensive nested tool learning evaluations. NesTools comprises a novel automatic data generation method to construct large-scale nested tool calls with different nesting structures. With manual review and refinement, the dataset is in high quality and closely aligned with real-world scenarios. Therefore, NesTools can serve as a new benchmark to evaluate the nested tool learning abilities of LLMs. We conduct extensive experiments on 22 LLMs, and provide in-depth analyses with NesTools, which shows that current LLMs still suffer from the complex nested tool learning task. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_11805 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | NesTools: A Dataset for Evaluating Nested Tool Learning Abilities of Large Language Models Han, Han Zhu, Tong Zhang, Xiang Wu, Mengsong Xiong, Hao Chen, Wenliang Computation and Language Large language models (LLMs) combined with tool learning have gained impressive results in real-world applications. During tool learning, LLMs may call multiple tools in nested orders, where the latter tool call may take the former response as its input parameters. However, current research on the nested tool learning capabilities is still under-explored, since the existing benchmarks lack relevant data instances. To address this problem, we introduce NesTools to bridge the current gap in comprehensive nested tool learning evaluations. NesTools comprises a novel automatic data generation method to construct large-scale nested tool calls with different nesting structures. With manual review and refinement, the dataset is in high quality and closely aligned with real-world scenarios. Therefore, NesTools can serve as a new benchmark to evaluate the nested tool learning abilities of LLMs. We conduct extensive experiments on 22 LLMs, and provide in-depth analyses with NesTools, which shows that current LLMs still suffer from the complex nested tool learning task. |
| title | NesTools: A Dataset for Evaluating Nested Tool Learning Abilities of Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.11805 |