LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917195876925440 |
|---|---|
| author | Li, Hao Zhang, Yiqun Guo, Zhaoyan Wang, Chenxu Tang, Shengji Zhang, Qiaosheng Chen, Yang Qi, Biqing Ye, Peng Bai, Lei Wang, Zhen Hu, Shuyue |
| author_facet | Li, Hao Zhang, Yiqun Guo, Zhaoyan Wang, Chenxu Tang, Shengji Zhang, Qiaosheng Chen, Yang Qi, Biqing Ye, Peng Bai, Lei Wang, Zhen Hu, Shuyue |
| contents | Large language model (LLM) routing assigns each query to the most suitable model from an ensemble. We introduce LLMRouterBench, a large-scale benchmark and unified framework for LLM routing. It comprises over 400K instances from 21 datasets and 33 models. Moreover, it provides comprehensive metrics for both performance-oriented routing and performance-cost trade-off routing, and integrates 10 representative routing baselines. Using LLMRouterBench, we systematically re-evaluate the field. While confirming strong model complementarity-the central premise of LLM routing-we find that many routing methods exhibit similar performance under unified evaluation, and several recent approaches, including commercial routers, fail to reliably outperform a simple baseline. Meanwhile, a substantial gap remains to the Oracle, driven primarily by persistent model-recall failures. We further show that backbone embedding models have limited impact, that larger ensembles exhibit diminishing returns compared to careful model curation, and that the benchmark also enables latency-aware analysis. All code and data are available at https://github.com/ynulihao/LLMRouterBench. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_07206 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing Li, Hao Zhang, Yiqun Guo, Zhaoyan Wang, Chenxu Tang, Shengji Zhang, Qiaosheng Chen, Yang Qi, Biqing Ye, Peng Bai, Lei Wang, Zhen Hu, Shuyue Artificial Intelligence Large language model (LLM) routing assigns each query to the most suitable model from an ensemble. We introduce LLMRouterBench, a large-scale benchmark and unified framework for LLM routing. It comprises over 400K instances from 21 datasets and 33 models. Moreover, it provides comprehensive metrics for both performance-oriented routing and performance-cost trade-off routing, and integrates 10 representative routing baselines. Using LLMRouterBench, we systematically re-evaluate the field. While confirming strong model complementarity-the central premise of LLM routing-we find that many routing methods exhibit similar performance under unified evaluation, and several recent approaches, including commercial routers, fail to reliably outperform a simple baseline. Meanwhile, a substantial gap remains to the Oracle, driven primarily by persistent model-recall failures. We further show that backbone embedding models have limited impact, that larger ensembles exhibit diminishing returns compared to careful model curation, and that the benchmark also enables latency-aware analysis. All code and data are available at https://github.com/ynulihao/LLMRouterBench. |
| title | LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2601.07206 |