A Framework for Testing and Adapting REST APIs as LLM Tools
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908534090760192 |
|---|---|
| author | Bandlamudi, Jayachandu Chaudhuri, Ritwik Gantayat, Neelamadhav Ghosh, Sambit Mukherjee, Kushal Agarwal, Prerna Sindhgatta, Renuka Mehta, Sameep |
| author_facet | Bandlamudi, Jayachandu Chaudhuri, Ritwik Gantayat, Neelamadhav Ghosh, Sambit Mukherjee, Kushal Agarwal, Prerna Sindhgatta, Renuka Mehta, Sameep |
| contents | Large Language Models (LLMs) are increasingly used to build autonomous agents that perform complex tasks with external tools, often exposed through APIs in enterprise systems. Direct use of these APIs is difficult due to the complex input schema and verbose responses. Current benchmarks overlook these challenges, leaving a gap in assessing API readiness for agent-driven automation. We present a testing framework that systematically evaluates enterprise APIs when wrapped as Python tools for LLM-based agents. The framework generates data-aware test cases, translates them into natural language instructions, and evaluates whether agents can correctly invoke the tool, handle their inputs, and process its responses. We apply the framework to generate over 2400 test cases across different domains and develop a taxonomy of common errors, including input misinterpretation, output failures, and schema mismatches. We further classify errors to support debugging and tool refinement. Our framework provides a systematic approach to enabling enterprise APIs as reliable tools for agent-based applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_15546 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Framework for Testing and Adapting REST APIs as LLM Tools Bandlamudi, Jayachandu Chaudhuri, Ritwik Gantayat, Neelamadhav Ghosh, Sambit Mukherjee, Kushal Agarwal, Prerna Sindhgatta, Renuka Mehta, Sameep Software Engineering Artificial Intelligence I.2.7 Large Language Models (LLMs) are increasingly used to build autonomous agents that perform complex tasks with external tools, often exposed through APIs in enterprise systems. Direct use of these APIs is difficult due to the complex input schema and verbose responses. Current benchmarks overlook these challenges, leaving a gap in assessing API readiness for agent-driven automation. We present a testing framework that systematically evaluates enterprise APIs when wrapped as Python tools for LLM-based agents. The framework generates data-aware test cases, translates them into natural language instructions, and evaluates whether agents can correctly invoke the tool, handle their inputs, and process its responses. We apply the framework to generate over 2400 test cases across different domains and develop a taxonomy of common errors, including input misinterpretation, output failures, and schema mismatches. We further classify errors to support debugging and tool refinement. Our framework provides a systematic approach to enabling enterprise APIs as reliable tools for agent-based applications. |
| title | A Framework for Testing and Adapting REST APIs as LLM Tools |
| topic | Software Engineering Artificial Intelligence I.2.7 |
| url | https://arxiv.org/abs/2504.15546 |