Agentic Plan Caching: Test-Time Memory for Fast and Cost-Efficient LLM Agents
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911400445607936 |
|---|---|
| author | Zhang, Qizheng Wornow, Michael Wan, Gerry Olukotun, Kunle |
| author_facet | Zhang, Qizheng Wornow, Michael Wan, Gerry Olukotun, Kunle |
| contents | LLM-based agent applications have shown increasingly remarkable capabilities in complex workflows but incur substantial costs and latency due to extensive planning and reasoning requirements. Existing LLM caching techniques (like context caching and semantic caching), primarily designed for serving chatbots, are insufficient for agent applications where outputs depend on external data and environmental contexts. We propose Agentic Plan Caching (APC), a novel test-time memory that extracts, stores, adapts, and reuses structured plan templates from planning stages of agent applications across semantically similar tasks to reduce the cost and latency of serving. Unlike traditional semantic caching, our system extracts plan templates from completed agent executions at test-time, employs keyword extraction to match new requests against cached plans, and utilizes lightweight models to adapt these templates to task-specific plans with contexts. Evaluation across multiple real-world agent applications shows that our system can reduce costs by 50.31% and latency by 27.28% on average while maintaining performance, offering a more efficient solution for serving LLM-based agents that complements existing LLM serving infrastructures. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_14852 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Agentic Plan Caching: Test-Time Memory for Fast and Cost-Efficient LLM Agents Zhang, Qizheng Wornow, Michael Wan, Gerry Olukotun, Kunle Distributed, Parallel, and Cluster Computing Artificial Intelligence Computation and Language Machine Learning Performance LLM-based agent applications have shown increasingly remarkable capabilities in complex workflows but incur substantial costs and latency due to extensive planning and reasoning requirements. Existing LLM caching techniques (like context caching and semantic caching), primarily designed for serving chatbots, are insufficient for agent applications where outputs depend on external data and environmental contexts. We propose Agentic Plan Caching (APC), a novel test-time memory that extracts, stores, adapts, and reuses structured plan templates from planning stages of agent applications across semantically similar tasks to reduce the cost and latency of serving. Unlike traditional semantic caching, our system extracts plan templates from completed agent executions at test-time, employs keyword extraction to match new requests against cached plans, and utilizes lightweight models to adapt these templates to task-specific plans with contexts. Evaluation across multiple real-world agent applications shows that our system can reduce costs by 50.31% and latency by 27.28% on average while maintaining performance, offering a more efficient solution for serving LLM-based agents that complements existing LLM serving infrastructures. |
| title | Agentic Plan Caching: Test-Time Memory for Fast and Cost-Efficient LLM Agents |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence Computation and Language Machine Learning Performance |
| url | https://arxiv.org/abs/2506.14852 |