Universal Model Routing for Efficient LLM Inference
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911070315085824 |
|---|---|
| author | Jitkrittum, Wittawat Narasimhan, Harikrishna Rawat, Ankit Singh Juneja, Jeevesh Wang, Congchao Wang, Zifeng Go, Alec Lee, Chen-Yu Shenoy, Pradeep Panigrahy, Rina Menon, Aditya Krishna Kumar, Sanjiv |
| author_facet | Jitkrittum, Wittawat Narasimhan, Harikrishna Rawat, Ankit Singh Juneja, Jeevesh Wang, Congchao Wang, Zifeng Go, Alec Lee, Chen-Yu Shenoy, Pradeep Panigrahy, Rina Menon, Aditya Krishna Kumar, Sanjiv |
| contents | Model routing is a simple technique for reducing the inference cost of large language models (LLMs), wherein one maintains a pool of candidate LLMs, and learns to route each prompt to the smallest feasible LLM. Existing works focus on learning a router for a fixed pool of LLMs. In this paper, we consider the problem of dynamic routing, where new, previously unobserved LLMs are available at test time. We propose UniRoute, a new approach to this problem that relies on representing each LLM as a feature vector, derived based on predictions on a set of representative prompts. Based on this, we detail two effective instantiations of UniRoute, relying on cluster-based routing and a learned cluster map respectively. We show that these are estimates of a theoretically optimal routing rule, and quantify their errors via an excess risk bound. Experiments on a range of public benchmarks show the effectiveness of UniRoute in routing amongst more than 30 unseen LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_08773 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Universal Model Routing for Efficient LLM Inference Jitkrittum, Wittawat Narasimhan, Harikrishna Rawat, Ankit Singh Juneja, Jeevesh Wang, Congchao Wang, Zifeng Go, Alec Lee, Chen-Yu Shenoy, Pradeep Panigrahy, Rina Menon, Aditya Krishna Kumar, Sanjiv Computation and Language Machine Learning Model routing is a simple technique for reducing the inference cost of large language models (LLMs), wherein one maintains a pool of candidate LLMs, and learns to route each prompt to the smallest feasible LLM. Existing works focus on learning a router for a fixed pool of LLMs. In this paper, we consider the problem of dynamic routing, where new, previously unobserved LLMs are available at test time. We propose UniRoute, a new approach to this problem that relies on representing each LLM as a feature vector, derived based on predictions on a set of representative prompts. Based on this, we detail two effective instantiations of UniRoute, relying on cluster-based routing and a learned cluster map respectively. We show that these are estimates of a theoretically optimal routing rule, and quantify their errors via an excess risk bound. Experiments on a range of public benchmarks show the effectiveness of UniRoute in routing amongst more than 30 unseen LLMs. |
| title | Universal Model Routing for Efficient LLM Inference |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2502.08773 |