LLM-dCache: Improving Tool-Augmented LLMs with GPT-Driven Localized Data Caching
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916405161492480 |
|---|---|
| author | Singh, Simranjit Fore, Michael Karatzas, Andreas Lee, Chaehong Jian, Yanan Shangguan, Longfei Yu, Fuxun Anagnostopoulos, Iraklis Stamoulis, Dimitrios |
| author_facet | Singh, Simranjit Fore, Michael Karatzas, Andreas Lee, Chaehong Jian, Yanan Shangguan, Longfei Yu, Fuxun Anagnostopoulos, Iraklis Stamoulis, Dimitrios |
| contents | As Large Language Models (LLMs) broaden their capabilities to manage thousands of API calls, they are confronted with complex data operations across vast datasets with significant overhead to the underlying system. In this work, we introduce LLM-dCache to optimize data accesses by treating cache operations as callable API functions exposed to the tool-augmented agent. We grant LLMs the autonomy to manage cache decisions via prompting, seamlessly integrating with existing function-calling mechanisms. Tested on an industry-scale massively parallel platform that spans hundreds of GPT endpoints and terabytes of imagery, our method improves Copilot times by an average of 1.24x across various LLMs and prompting techniques. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_06799 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LLM-dCache: Improving Tool-Augmented LLMs with GPT-Driven Localized Data Caching Singh, Simranjit Fore, Michael Karatzas, Andreas Lee, Chaehong Jian, Yanan Shangguan, Longfei Yu, Fuxun Anagnostopoulos, Iraklis Stamoulis, Dimitrios Distributed, Parallel, and Cluster Computing Computation and Language As Large Language Models (LLMs) broaden their capabilities to manage thousands of API calls, they are confronted with complex data operations across vast datasets with significant overhead to the underlying system. In this work, we introduce LLM-dCache to optimize data accesses by treating cache operations as callable API functions exposed to the tool-augmented agent. We grant LLMs the autonomy to manage cache decisions via prompting, seamlessly integrating with existing function-calling mechanisms. Tested on an industry-scale massively parallel platform that spans hundreds of GPT endpoints and terabytes of imagery, our method improves Copilot times by an average of 1.24x across various LLMs and prompting techniques. |
| title | LLM-dCache: Improving Tool-Augmented LLMs with GPT-Driven Localized Data Caching |
| topic | Distributed, Parallel, and Cluster Computing Computation and Language |
| url | https://arxiv.org/abs/2406.06799 |