Routing with Generated Data: Annotation-Free LLM Skill Estimation and Expert Selection
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908765494706176 |
|---|---|
| author | Niu, Tianyi Chen, Justin Chih-Yao Winata, Genta Indra Zhang, Shi-Xiong Chakraborty, Supriyo Sahu, Sambit Zhang, Yue Stengel-Eskin, Elias Bansal, Mohit |
| author_facet | Niu, Tianyi Chen, Justin Chih-Yao Winata, Genta Indra Zhang, Shi-Xiong Chakraborty, Supriyo Sahu, Sambit Zhang, Yue Stengel-Eskin, Elias Bansal, Mohit |
| contents | Large Language Model (LLM) routers dynamically select optimal models for given inputs. Existing approaches typically assume access to ground-truth labeled data, which is often unavailable in practice, especially when user request distributions are heterogeneous and unknown. We introduce Routing with Generated Data (RGD), a challenging setting in which routers are trained exclusively on generated queries and answers produced from high-level task descriptions by generator LLMs. We evaluate query-answer routers (using both queries and labels) and query-only routers across four diverse benchmarks and 12 models, finding that query-answer routers degrade faster than query-only routers as generator quality decreases. Our analysis reveals two crucial characteristics of effective generators: they must accurately respond to their own questions, and their questions must produce sufficient performance differentiation among the model pool. We then show how filtering for these characteristics can improve the quality of generated data. We further propose CASCAL, a novel query-only router that estimates model correctness through consensus voting and identifies model-specific skill niches via hierarchical clustering. CASCAL is substantially more robust to generator quality, outperforming the best query-answer router by 4.6% absolute accuracy when trained on weak generator data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_09692 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Routing with Generated Data: Annotation-Free LLM Skill Estimation and Expert Selection Niu, Tianyi Chen, Justin Chih-Yao Winata, Genta Indra Zhang, Shi-Xiong Chakraborty, Supriyo Sahu, Sambit Zhang, Yue Stengel-Eskin, Elias Bansal, Mohit Computation and Language Artificial Intelligence Machine Learning Large Language Model (LLM) routers dynamically select optimal models for given inputs. Existing approaches typically assume access to ground-truth labeled data, which is often unavailable in practice, especially when user request distributions are heterogeneous and unknown. We introduce Routing with Generated Data (RGD), a challenging setting in which routers are trained exclusively on generated queries and answers produced from high-level task descriptions by generator LLMs. We evaluate query-answer routers (using both queries and labels) and query-only routers across four diverse benchmarks and 12 models, finding that query-answer routers degrade faster than query-only routers as generator quality decreases. Our analysis reveals two crucial characteristics of effective generators: they must accurately respond to their own questions, and their questions must produce sufficient performance differentiation among the model pool. We then show how filtering for these characteristics can improve the quality of generated data. We further propose CASCAL, a novel query-only router that estimates model correctness through consensus voting and identifies model-specific skill niches via hierarchical clustering. CASCAL is substantially more robust to generator quality, outperforming the best query-answer router by 4.6% absolute accuracy when trained on weak generator data. |
| title | Routing with Generated Data: Annotation-Free LLM Skill Estimation and Expert Selection |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2601.09692 |