Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910418507661312 |
|---|---|
| author | Ding, Dujian Mallick, Ankur Wang, Chi Sim, Robert Mukherjee, Subhabrata Ruhle, Victor Lakshmanan, Laks V. S. Awadallah, Ahmed Hassan |
| author_facet | Ding, Dujian Mallick, Ankur Wang, Chi Sim, Robert Mukherjee, Subhabrata Ruhle, Victor Lakshmanan, Laks V. S. Awadallah, Ahmed Hassan |
| contents | Large language models (LLMs) excel in most NLP tasks but also require expensive cloud servers for deployment due to their size, while smaller models that can be deployed on lower cost (e.g., edge) devices, tend to lag behind in terms of response quality. Therefore in this work we propose a hybrid inference approach which combines their respective strengths to save cost and maintain quality. Our approach uses a router that assigns queries to the small or large model based on the predicted query difficulty and the desired quality level. The desired quality level can be tuned dynamically at test time to seamlessly trade quality for cost as per the scenario requirements. In experiments our approach allows us to make up to 40% fewer calls to the large model, with no drop in response quality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_14618 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing Ding, Dujian Mallick, Ankur Wang, Chi Sim, Robert Mukherjee, Subhabrata Ruhle, Victor Lakshmanan, Laks V. S. Awadallah, Ahmed Hassan Machine Learning Artificial Intelligence Computation and Language Large language models (LLMs) excel in most NLP tasks but also require expensive cloud servers for deployment due to their size, while smaller models that can be deployed on lower cost (e.g., edge) devices, tend to lag behind in terms of response quality. Therefore in this work we propose a hybrid inference approach which combines their respective strengths to save cost and maintain quality. Our approach uses a router that assigns queries to the small or large model based on the predicted query difficulty and the desired quality level. The desired quality level can be tuned dynamically at test time to seamlessly trade quality for cost as per the scenario requirements. In experiments our approach allows us to make up to 40% fewer calls to the large model, with no drop in response quality. |
| title | Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2404.14618 |