ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenario
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913654706798592 |
|---|---|
| author | Zhong, Lucen Du, Zhengxiao Zhang, Xiaohan Hu, Haiyi Tang, Jie |
| author_facet | Zhong, Lucen Du, Zhengxiao Zhang, Xiaohan Hu, Haiyi Tang, Jie |
| contents | Enhancing large language models (LLMs) with real-time APIs can help generate more accurate and up-to-date responses. However, evaluating the function calling abilities of LLMs in real-world scenarios remains under-explored due to the complexity of data collection and evaluation. In this work, we introduce ComplexFuncBench, a benchmark for complex function calling across five real-world scenarios. Compared to existing benchmarks, ComplexFuncBench encompasses multi-step and constrained function calling, which requires long-parameter filing, parameter value reasoning, and 128k long context. Additionally, we propose an automatic framework, ComplexEval, for quantitatively evaluating complex function calling tasks. Through comprehensive experiments, we demonstrate the deficiencies of state-of-the-art LLMs in function calling and suggest future directions for optimizing these capabilities. The data and code are available at \url{https://github.com/THUDM/ComplexFuncBench}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_10132 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenario Zhong, Lucen Du, Zhengxiao Zhang, Xiaohan Hu, Haiyi Tang, Jie Computation and Language Enhancing large language models (LLMs) with real-time APIs can help generate more accurate and up-to-date responses. However, evaluating the function calling abilities of LLMs in real-world scenarios remains under-explored due to the complexity of data collection and evaluation. In this work, we introduce ComplexFuncBench, a benchmark for complex function calling across five real-world scenarios. Compared to existing benchmarks, ComplexFuncBench encompasses multi-step and constrained function calling, which requires long-parameter filing, parameter value reasoning, and 128k long context. Additionally, we propose an automatic framework, ComplexEval, for quantitatively evaluating complex function calling tasks. Through comprehensive experiments, we demonstrate the deficiencies of state-of-the-art LLMs in function calling and suggest future directions for optimizing these capabilities. The data and code are available at \url{https://github.com/THUDM/ComplexFuncBench}. |
| title | ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenario |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2501.10132 |