vLLM Semantic Router: Signal Driven Decision Routing for Mixture-of-Modality Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912947246202880 |
|---|---|
| author | Liu, Xunzhuo Chen, Huamin Lu, Samzong Ovadia, Yossi Wen, Guohong Wu, Hao Tan, Zhengda Zhang, Jintao Zedan, Senan Kerido, Yehudit Weiss, Liav Zhang, Haichen Yu, Bishen Balum, Asaad Limoy, Noa Samara, Abdallah Fan, Baofa Salisbury, Brent Cook, Ryan Wang, Zhijie Pan, Qiping Khan, Rehan Goswami, Avishek Zhang, Houston H. Wang, Shuyi Tang, Ziang Han, Fang Hassan, Zohaib Zheng, Jianqiao Changrani, Avinash |
| author_facet | Liu, Xunzhuo Chen, Huamin Lu, Samzong Ovadia, Yossi Wen, Guohong Wu, Hao Tan, Zhengda Zhang, Jintao Zedan, Senan Kerido, Yehudit Weiss, Liav Zhang, Haichen Yu, Bishen Balum, Asaad Limoy, Noa Samara, Abdallah Fan, Baofa Salisbury, Brent Cook, Ryan Wang, Zhijie Pan, Qiping Khan, Rehan Goswami, Avishek Zhang, Houston H. Wang, Shuyi Tang, Ziang Han, Fang Hassan, Zohaib Zheng, Jianqiao Changrani, Avinash |
| contents | As large language models (LLMs) diversify across modalities, capabilities, and cost profiles, the problem of intelligent request routing -- selecting the right model for each query at inference time -- has become a critical systems challenge. We present vLLM Semantic Router, a signal-driven decision routing framework for Mixture-of-Modality (MoM) model deployments.
The central innovation is composable signal orchestration: the system extracts heterogeneous signal types from each request -- from sub-millisecond heuristic features (keyword patterns, language detection, context length, role-based authorization) to neural classifiers (domain, embedding similarity, factual grounding, modality) -- and composes them through configurable Boolean decision rules into deployment-specific routing policies. Different deployment scenarios -- multi-cloud enterprise, privacy-regulated, cost-optimized, latency-sensitive -- are expressed as different signal-decision configurations over the same architecture, without code changes.
Matched decisions drive semantic model routing: over a dozen of selection algorithms analyze request characteristics to find the best model cost-effectively, while per-decision plugin chains enforce privacy and safety constraints (jailbreak detection, PII filtering, hallucination detection via the three-stage HaluGate pipeline).
The system provides OpenAI API support for stateful multi-turn conversations, multi-endpoint and multi-provider routing across heterogeneous backends (vLLM, OpenAI, Anthropic, Azure, Bedrock, Gemini, Vertex AI), and a pluggable authorization factory supporting multiple auth providers. Deployed in production as an Envoy external processor, the architecture demonstrates that composable signal orchestration enables a single routing framework to serve diverse deployment scenarios with differentiated cost, privacy, and safety policies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_04444 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | vLLM Semantic Router: Signal Driven Decision Routing for Mixture-of-Modality Models Liu, Xunzhuo Chen, Huamin Lu, Samzong Ovadia, Yossi Wen, Guohong Wu, Hao Tan, Zhengda Zhang, Jintao Zedan, Senan Kerido, Yehudit Weiss, Liav Zhang, Haichen Yu, Bishen Balum, Asaad Limoy, Noa Samara, Abdallah Fan, Baofa Salisbury, Brent Cook, Ryan Wang, Zhijie Pan, Qiping Khan, Rehan Goswami, Avishek Zhang, Houston H. Wang, Shuyi Tang, Ziang Han, Fang Hassan, Zohaib Zheng, Jianqiao Changrani, Avinash Networking and Internet Architecture Artificial Intelligence As large language models (LLMs) diversify across modalities, capabilities, and cost profiles, the problem of intelligent request routing -- selecting the right model for each query at inference time -- has become a critical systems challenge. We present vLLM Semantic Router, a signal-driven decision routing framework for Mixture-of-Modality (MoM) model deployments. The central innovation is composable signal orchestration: the system extracts heterogeneous signal types from each request -- from sub-millisecond heuristic features (keyword patterns, language detection, context length, role-based authorization) to neural classifiers (domain, embedding similarity, factual grounding, modality) -- and composes them through configurable Boolean decision rules into deployment-specific routing policies. Different deployment scenarios -- multi-cloud enterprise, privacy-regulated, cost-optimized, latency-sensitive -- are expressed as different signal-decision configurations over the same architecture, without code changes. Matched decisions drive semantic model routing: over a dozen of selection algorithms analyze request characteristics to find the best model cost-effectively, while per-decision plugin chains enforce privacy and safety constraints (jailbreak detection, PII filtering, hallucination detection via the three-stage HaluGate pipeline). The system provides OpenAI API support for stateful multi-turn conversations, multi-endpoint and multi-provider routing across heterogeneous backends (vLLM, OpenAI, Anthropic, Azure, Bedrock, Gemini, Vertex AI), and a pluggable authorization factory supporting multiple auth providers. Deployed in production as an Envoy external processor, the architecture demonstrates that composable signal orchestration enables a single routing framework to serve diverse deployment scenarios with differentiated cost, privacy, and safety policies. |
| title | vLLM Semantic Router: Signal Driven Decision Routing for Mixture-of-Modality Models |
| topic | Networking and Internet Architecture Artificial Intelligence |
| url | https://arxiv.org/abs/2603.04444 |