Optimizing Agentic Language Model Inference via Speculative Tool Calls
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909968386490368 |
|---|---|
| author | Nichols, Daniel Singhania, Prajwal Jekel, Charles Bhatele, Abhinav Menon, Harshitha |
| author_facet | Nichols, Daniel Singhania, Prajwal Jekel, Charles Bhatele, Abhinav Menon, Harshitha |
| contents | Language models (LMs) are becoming increasingly dependent on external tools. LM-based agentic frameworks frequently interact with their environment via such tools to search files, run code, call APIs, etc. Further, modern reasoning-based LMs use tools such as web search and Python code execution to enhance their reasoning capabilities. While tools greatly improve the capabilities of LMs, they also introduce performance bottlenecks during the inference process. In this paper, we introduce novel systems optimizations to address such performance bottlenecks by speculating tool calls and forcing sequences to remain resident in the inference engine to minimize overheads. Our optimizations lead to throughput improvements of several hundred tokens per second when hosting inference for LM agents. We provide a theoretical analysis of our algorithms to provide insights into speculation configurations that will yield the best performance. Further, we recommend a new "tool cache" API endpoint to enable LM providers to easily adopt these optimizations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_15834 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Optimizing Agentic Language Model Inference via Speculative Tool Calls Nichols, Daniel Singhania, Prajwal Jekel, Charles Bhatele, Abhinav Menon, Harshitha Programming Languages Artificial Intelligence Distributed, Parallel, and Cluster Computing Performance Software Engineering Language models (LMs) are becoming increasingly dependent on external tools. LM-based agentic frameworks frequently interact with their environment via such tools to search files, run code, call APIs, etc. Further, modern reasoning-based LMs use tools such as web search and Python code execution to enhance their reasoning capabilities. While tools greatly improve the capabilities of LMs, they also introduce performance bottlenecks during the inference process. In this paper, we introduce novel systems optimizations to address such performance bottlenecks by speculating tool calls and forcing sequences to remain resident in the inference engine to minimize overheads. Our optimizations lead to throughput improvements of several hundred tokens per second when hosting inference for LM agents. We provide a theoretical analysis of our algorithms to provide insights into speculation configurations that will yield the best performance. Further, we recommend a new "tool cache" API endpoint to enable LM providers to easily adopt these optimizations. |
| title | Optimizing Agentic Language Model Inference via Speculative Tool Calls |
| topic | Programming Languages Artificial Intelligence Distributed, Parallel, and Cluster Computing Performance Software Engineering |
| url | https://arxiv.org/abs/2512.15834 |