StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909525037023232 |
|---|---|
| author | Guo, Zhicheng Cheng, Sijie Wang, Hao Liang, Shihao Qin, Yujia Li, Peng Liu, Zhiyuan Sun, Maosong Liu, Yang |
| author_facet | Guo, Zhicheng Cheng, Sijie Wang, Hao Liang, Shihao Qin, Yujia Li, Peng Liu, Zhiyuan Sun, Maosong Liu, Yang |
| contents | Large Language Models (LLMs) have witnessed remarkable advancements in recent years, prompting the exploration of tool learning, which integrates LLMs with external tools to address diverse real-world challenges. Assessing the capability of LLMs to utilise tools necessitates large-scale and stable benchmarks. However, previous works relied on either hand-crafted online tools with limited scale, or large-scale real online APIs suffering from instability of API status. To address this problem, we introduce StableToolBench, a benchmark evolving from ToolBench, proposing a virtual API server and stable evaluation system. The virtual API server contains a caching system and API simulators which are complementary to alleviate the change in API status. Meanwhile, the stable evaluation system designs solvable pass and win rates using GPT-4 as the automatic evaluator to eliminate the randomness during evaluation. Experimental results demonstrate the stability of StableToolBench, and further discuss the effectiveness of API simulators, the caching system, and the evaluator system. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_07714 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models Guo, Zhicheng Cheng, Sijie Wang, Hao Liang, Shihao Qin, Yujia Li, Peng Liu, Zhiyuan Sun, Maosong Liu, Yang Computation and Language Large Language Models (LLMs) have witnessed remarkable advancements in recent years, prompting the exploration of tool learning, which integrates LLMs with external tools to address diverse real-world challenges. Assessing the capability of LLMs to utilise tools necessitates large-scale and stable benchmarks. However, previous works relied on either hand-crafted online tools with limited scale, or large-scale real online APIs suffering from instability of API status. To address this problem, we introduce StableToolBench, a benchmark evolving from ToolBench, proposing a virtual API server and stable evaluation system. The virtual API server contains a caching system and API simulators which are complementary to alleviate the change in API status. Meanwhile, the stable evaluation system designs solvable pass and win rates using GPT-4 as the automatic evaluator to eliminate the randomness during evaluation. Experimental results demonstrate the stability of StableToolBench, and further discuss the effectiveness of API simulators, the caching system, and the evaluator system. |
| title | StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2403.07714 |