Cache & Distil: Optimising API Calls to Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912345376161792 |
|---|---|
| author | Ramírez, Guillem Lindemann, Matthias Birch, Alexandra Titov, Ivan |
| author_facet | Ramírez, Guillem Lindemann, Matthias Birch, Alexandra Titov, Ivan |
| contents | Large-scale deployment of generative AI tools often depends on costly API calls to a Large Language Model (LLM) to fulfil user queries. To curtail the frequency of these calls, one can employ a smaller language model -- a student -- which is continuously trained on the responses of the LLM. This student gradually gains proficiency in independently handling an increasing number of user requests, a process we term neural caching. The crucial element in neural caching is a policy that decides which requests should be processed by the student alone and which should be redirected to the LLM, subsequently aiding the student's learning. In this study, we focus on classification tasks, and we consider a range of classic active learning-based selection criteria as the policy. Our experiments suggest that Margin Sampling and Query by Committee bring consistent benefits across tasks and budgets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_13561 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Cache & Distil: Optimising API Calls to Large Language Models Ramírez, Guillem Lindemann, Matthias Birch, Alexandra Titov, Ivan Computation and Language Machine Learning Large-scale deployment of generative AI tools often depends on costly API calls to a Large Language Model (LLM) to fulfil user queries. To curtail the frequency of these calls, one can employ a smaller language model -- a student -- which is continuously trained on the responses of the LLM. This student gradually gains proficiency in independently handling an increasing number of user requests, a process we term neural caching. The crucial element in neural caching is a policy that decides which requests should be processed by the student alone and which should be redirected to the LLM, subsequently aiding the student's learning. In this study, we focus on classification tasks, and we consider a range of classic active learning-based selection criteria as the policy. Our experiments suggest that Margin Sampling and Query by Committee bring consistent benefits across tasks and budgets. |
| title | Cache & Distil: Optimising API Calls to Large Language Models |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2310.13561 |