MESS+: Dynamically Learned Inference-Time LLM Routing in Model Zoos with Service Level Guarantees
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909866680909824 |
|---|---|
| author | Woisetschläger, Herbert Zhang, Ryan Wang, Shiqiang Jacobsen, Hans-Arno |
| author_facet | Woisetschläger, Herbert Zhang, Ryan Wang, Shiqiang Jacobsen, Hans-Arno |
| contents | Open-weight large language model (LLM) zoos provide access to numerous high-quality models, but selecting the appropriate model for specific tasks remains challenging and requires technical expertise. Most users simply want factually correct, safe, and satisfying responses without concerning themselves with model technicalities, while inference service providers prioritize minimizing operating costs. These competing interests are typically mediated through service level agreements (SLAs) that guarantee minimum service quality. We introduce MESS+, a stochastic optimization algorithm for cost-optimal LLM request routing while providing rigorous SLA compliance guarantees. MESS+ learns request satisfaction probabilities of LLMs in real-time as users interact with the system, based on which model selection decisions are made by solving a per-request optimization problem. Our algorithm includes a novel combination of virtual queues and request satisfaction prediction, along with a theoretical analysis of cost optimality and constraint satisfaction. Across a wide range of state-of-the-art LLM benchmarks, MESS+ achieves an average of $2\times$ cost savings compared to existing LLM routing techniques. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_19947 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MESS+: Dynamically Learned Inference-Time LLM Routing in Model Zoos with Service Level Guarantees Woisetschläger, Herbert Zhang, Ryan Wang, Shiqiang Jacobsen, Hans-Arno Machine Learning Artificial Intelligence Systems and Control I.2; I.2.7; I.2.8 Open-weight large language model (LLM) zoos provide access to numerous high-quality models, but selecting the appropriate model for specific tasks remains challenging and requires technical expertise. Most users simply want factually correct, safe, and satisfying responses without concerning themselves with model technicalities, while inference service providers prioritize minimizing operating costs. These competing interests are typically mediated through service level agreements (SLAs) that guarantee minimum service quality. We introduce MESS+, a stochastic optimization algorithm for cost-optimal LLM request routing while providing rigorous SLA compliance guarantees. MESS+ learns request satisfaction probabilities of LLMs in real-time as users interact with the system, based on which model selection decisions are made by solving a per-request optimization problem. Our algorithm includes a novel combination of virtual queues and request satisfaction prediction, along with a theoretical analysis of cost optimality and constraint satisfaction. Across a wide range of state-of-the-art LLM benchmarks, MESS+ achieves an average of $2\times$ cost savings compared to existing LLM routing techniques. |
| title | MESS+: Dynamically Learned Inference-Time LLM Routing in Model Zoos with Service Level Guarantees |
| topic | Machine Learning Artificial Intelligence Systems and Control I.2; I.2.7; I.2.8 |
| url | https://arxiv.org/abs/2505.19947 |