Robust Batch-Level Query Routing for Large Language Models under Cost and Capacity Constraints
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912985516081152 |
|---|---|
| author | Markovic-Voronov, Jelena Behdin, Kayhan Xu, Yuanda Zhou, Zhengze Wang, Zhipeng Mazumder, Rahul |
| author_facet | Markovic-Voronov, Jelena Behdin, Kayhan Xu, Yuanda Zhou, Zhengze Wang, Zhipeng Mazumder, Rahul |
| contents | We study the problem of routing queries to large language models (LLMs) under cost, GPU resources, and concurrency constraints. Prior per-query routing methods often fail to control batch-level cost, especially under non-uniform or adversarial batching. To address this, we propose a batch-level, resource-aware routing framework that jointly optimizes model assignment for each batch while respecting cost and model capacity limits. We further introduce a robust variant that accounts for uncertainty in predicted LLM performance, along with an offline instance allocation procedure that balances quality and throughput across multiple models. Experiments on two multi-task LLM benchmarks show that robustness improves accuracy by 1-14% over non-robust counterparts (depending on the performance estimator), batch-level routing outperforms per-query methods by up to 24% under adversarial batching, and optimized instance allocation yields additional gains of up to 3% compared to a non-optimized allocation, all while strictly controlling cost and GPU resource constraints. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_26796 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Robust Batch-Level Query Routing for Large Language Models under Cost and Capacity Constraints Markovic-Voronov, Jelena Behdin, Kayhan Xu, Yuanda Zhou, Zhengze Wang, Zhipeng Mazumder, Rahul Machine Learning Artificial Intelligence We study the problem of routing queries to large language models (LLMs) under cost, GPU resources, and concurrency constraints. Prior per-query routing methods often fail to control batch-level cost, especially under non-uniform or adversarial batching. To address this, we propose a batch-level, resource-aware routing framework that jointly optimizes model assignment for each batch while respecting cost and model capacity limits. We further introduce a robust variant that accounts for uncertainty in predicted LLM performance, along with an offline instance allocation procedure that balances quality and throughput across multiple models. Experiments on two multi-task LLM benchmarks show that robustness improves accuracy by 1-14% over non-robust counterparts (depending on the performance estimator), batch-level routing outperforms per-query methods by up to 24% under adversarial batching, and optimized instance allocation yields additional gains of up to 3% compared to a non-optimized allocation, all while strictly controlling cost and GPU resource constraints. |
| title | Robust Batch-Level Query Routing for Large Language Models under Cost and Capacity Constraints |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2603.26796 |