Towards Fair and Comprehensive Evaluation of Routers in Collaborative LLM Systems
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917270611034112 |
|---|---|
| author | Wu, Wanxing Zhu, He Li, Yixia Yang, Lei Zhao, Jiehui Wang, Hongru Yang, Jian Wang, Benyou Jing, Bingyi Chen, Guanhua |
| author_facet | Wu, Wanxing Zhu, He Li, Yixia Yang, Lei Zhao, Jiehui Wang, Hongru Yang, Jian Wang, Benyou Jing, Bingyi Chen, Guanhua |
| contents | Large language models (LLMs) have achieved success, but cost and privacy constraints necessitate deploying smaller models locally while offloading complex queries to cloud-based models. Existing router evaluations are unsystematic, overlooking scenario-specific requirements and out-of-distribution robustness. We propose RouterXBench, a principled evaluation framework with three dimensions: router ability, scenario alignment, and cross-domain robustness. Unlike prior work that relies on output probabilities or external embeddings, we utilize internal hidden states that capture model uncertainty before answer generation. We introduce ProbeDirichlet, a lightweight router that aggregates cross-layer hidden states via learnable Dirichlet distributions with probabilistic training. Trained on multi-domain data, it generalizes robustly across in-domain and out-of-distribution scenarios. Our results show ProbeDirichlet achieves 16.68% and 18.86% relative improvements over the best baselines in router ability and high-accuracy scenarios, with consistent performance across model families, model scales, heterogeneous tasks, and agentic workflows. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_11877 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Towards Fair and Comprehensive Evaluation of Routers in Collaborative LLM Systems Wu, Wanxing Zhu, He Li, Yixia Yang, Lei Zhao, Jiehui Wang, Hongru Yang, Jian Wang, Benyou Jing, Bingyi Chen, Guanhua Computation and Language Artificial Intelligence Large language models (LLMs) have achieved success, but cost and privacy constraints necessitate deploying smaller models locally while offloading complex queries to cloud-based models. Existing router evaluations are unsystematic, overlooking scenario-specific requirements and out-of-distribution robustness. We propose RouterXBench, a principled evaluation framework with three dimensions: router ability, scenario alignment, and cross-domain robustness. Unlike prior work that relies on output probabilities or external embeddings, we utilize internal hidden states that capture model uncertainty before answer generation. We introduce ProbeDirichlet, a lightweight router that aggregates cross-layer hidden states via learnable Dirichlet distributions with probabilistic training. Trained on multi-domain data, it generalizes robustly across in-domain and out-of-distribution scenarios. Our results show ProbeDirichlet achieves 16.68% and 18.86% relative improvements over the best baselines in router ability and high-accuracy scenarios, with consistent performance across model families, model scales, heterogeneous tasks, and agentic workflows. |
| title | Towards Fair and Comprehensive Evaluation of Routers in Collaborative LLM Systems |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2602.11877 |