NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913849150537728 |
|---|---|
| author | Basu, Kinjal Abdelaziz, Ibrahim Kate, Kiran Agarwal, Mayank Crouse, Maxwell Rizk, Yara Bradford, Kelsey Munawar, Asim Kumaravel, Sadhana Goyal, Saurabh Wang, Xin Lastras, Luis A. Kapanipathi, Pavan |
| author_facet | Basu, Kinjal Abdelaziz, Ibrahim Kate, Kiran Agarwal, Mayank Crouse, Maxwell Rizk, Yara Bradford, Kelsey Munawar, Asim Kumaravel, Sadhana Goyal, Saurabh Wang, Xin Lastras, Luis A. Kapanipathi, Pavan |
| contents | The resurgence of autonomous agents built using large language models (LLMs) to solve complex real-world tasks has brought increased focus on LLMs' fundamental ability of tool or function calling. At the core of these agents, an LLM must plan, execute, and respond using external tools, APIs, and custom functions. Research on tool calling has gathered momentum, but evaluation benchmarks and datasets representing the complexity of the tasks have lagged behind. In this work, we focus on one such complexity, nested sequencing, with the goal of extending existing benchmarks and evaluation. Specifically, we present NESTFUL, a benchmark to evaluate LLMs on nested sequences of API calls, i.e., sequences where the output of one API call is passed as input to a subsequent call. NESTFUL contains 1800+ nested sequences where all the function calls are executable. Experimental results on a variety of models show that the best-performing model (GPT-4o) achieves a full sequence match accuracy of 28% and a win-rate of 60%, necessitating a large scope for improvement in the nested sequencing aspect of function calling. Our analysis of these results provides possible future research directions for the community, in addition to a benchmark to track progress. We have released the NESTFUL dataset under the Apache 2.0 license at https://github.com/IBM/NESTFUL. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_03797 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls Basu, Kinjal Abdelaziz, Ibrahim Kate, Kiran Agarwal, Mayank Crouse, Maxwell Rizk, Yara Bradford, Kelsey Munawar, Asim Kumaravel, Sadhana Goyal, Saurabh Wang, Xin Lastras, Luis A. Kapanipathi, Pavan Artificial Intelligence Computation and Language The resurgence of autonomous agents built using large language models (LLMs) to solve complex real-world tasks has brought increased focus on LLMs' fundamental ability of tool or function calling. At the core of these agents, an LLM must plan, execute, and respond using external tools, APIs, and custom functions. Research on tool calling has gathered momentum, but evaluation benchmarks and datasets representing the complexity of the tasks have lagged behind. In this work, we focus on one such complexity, nested sequencing, with the goal of extending existing benchmarks and evaluation. Specifically, we present NESTFUL, a benchmark to evaluate LLMs on nested sequences of API calls, i.e., sequences where the output of one API call is passed as input to a subsequent call. NESTFUL contains 1800+ nested sequences where all the function calls are executable. Experimental results on a variety of models show that the best-performing model (GPT-4o) achieves a full sequence match accuracy of 28% and a win-rate of 60%, necessitating a large scope for improvement in the nested sequencing aspect of function calling. Our analysis of these results provides possible future research directions for the community, in addition to a benchmark to track progress. We have released the NESTFUL dataset under the Apache 2.0 license at https://github.com/IBM/NESTFUL. |
| title | NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2409.03797 |