CallNavi, A Challenge and Empirical Study on LLM Function Calling and Routing
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916706187739136 |
|---|---|
| author | Song, Yewei Tang, Xunzhu Lothritz, Cedric Ezzini, Saad Klein, Jacques Bissyandé, Tegawendé F. Boytsov, Andrey Ble, Ulrick Goujon, Anne |
| author_facet | Song, Yewei Tang, Xunzhu Lothritz, Cedric Ezzini, Saad Klein, Jacques Bissyandé, Tegawendé F. Boytsov, Andrey Ble, Ulrick Goujon, Anne |
| contents | API-driven chatbot systems are increasingly integral to software engineering applications, yet their effectiveness hinges on accurately generating and executing API calls. This is particularly challenging in scenarios requiring multi-step interactions with complex parameterization and nested API dependencies. Addressing these challenges, this work contributes to the evaluation and assessment of AI-based software development through three key advancements: (1) the introduction of a novel dataset specifically designed for benchmarking API function selection, parameter generation, and nested API execution; (2) an empirical evaluation of state-of-the-art language models, analyzing their performance across varying task complexities in API function generation and parameter accuracy; and (3) a hybrid approach to API routing, combining general-purpose large language models for API selection with fine-tuned models and prompt engineering for parameter generation. These innovations significantly improve API execution in chatbot systems, offering practical methodologies for enhancing software design, testing, and operational workflows in real-world software engineering contexts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_05255 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CallNavi, A Challenge and Empirical Study on LLM Function Calling and Routing Song, Yewei Tang, Xunzhu Lothritz, Cedric Ezzini, Saad Klein, Jacques Bissyandé, Tegawendé F. Boytsov, Andrey Ble, Ulrick Goujon, Anne Software Engineering Computation and Language API-driven chatbot systems are increasingly integral to software engineering applications, yet their effectiveness hinges on accurately generating and executing API calls. This is particularly challenging in scenarios requiring multi-step interactions with complex parameterization and nested API dependencies. Addressing these challenges, this work contributes to the evaluation and assessment of AI-based software development through three key advancements: (1) the introduction of a novel dataset specifically designed for benchmarking API function selection, parameter generation, and nested API execution; (2) an empirical evaluation of state-of-the-art language models, analyzing their performance across varying task complexities in API function generation and parameter accuracy; and (3) a hybrid approach to API routing, combining general-purpose large language models for API selection with fine-tuned models and prompt engineering for parameter generation. These innovations significantly improve API execution in chatbot systems, offering practical methodologies for enhancing software design, testing, and operational workflows in real-world software engineering contexts. |
| title | CallNavi, A Challenge and Empirical Study on LLM Function Calling and Routing |
| topic | Software Engineering Computation and Language |
| url | https://arxiv.org/abs/2501.05255 |