Imagen de Portada

98$\times$ Faster LLM Routing Without a Dedicated GPU: Flash Attention, Prompt Compression, and Near-Streaming for the vLLM Semantic Router

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Liu, Xunzhuo, He, Bowei, Liu, Xue, Luo, Andy, Zhang, Haichen, Chen, Huamin
Formato:	Preprint
Publicado:	2026
Materias:	Computation and Language
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

When to Reason: Semantic Router for vLLM
por: Wang, Chen, et al.
Publicado: (2025)

The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Project
por: Chen, Huamin, et al.
Publicado: (2026)

Dual-Pool Token-Budget Routing for Cost-Efficient and Reliable LLM Serving
por: Liu, Xunzhuo, et al.
Publicado: (2026)

Outcome-Aware Tool Selection for Semantic Routers: Latency-Constrained Learning Without LLM Inference
por: Chen, Huamin, et al.
Publicado: (2026)

vLLM Semantic Router: Signal Driven Decision Routing for Mixture-of-Modality Models
por: Liu, Xunzhuo, et al.
Publicado: (2026)

Adaptive Vision-Language Model Routing for Computer Use Agents
por: Liu, Xunzhuo, et al.
Publicado: (2026)

Knowledge Access Beats Model Size: Memory Augmented Routing for Persistent AI Agents
por: Liu, Xunzhuo, et al.
Publicado: (2026)

FleetOpt: Analytical Fleet Provisioning for LLM Inference with Compress-and-Route as Implementation Mechanism
por: Chen, Huamin, et al.
Publicado: (2026)

The 1/W Law: An Analytical Study of Context-Length Routing Topology and GPU Generation Gains for LLM Inference Energy Efficiency
por: Chen, Huamin, et al.
Publicado: (2026)

Visual Confused Deputy: Exploiting and Defending Perception Failures in Computer-Using Agents
por: Liu, Xunzhuo, et al.
Publicado: (2026)

Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inference
por: Chen, Huamin, et al.
Publicado: (2026)

Conflict-Free Policy Languages for Probabilistic ML Predicates: A Framework and Case Study with the Semantic Router DSL
por: Liu, Xunzhuo, et al.
Publicado: (2026)

Fast and Faithful: Real-Time Verification for Long-Document Retrieval-Augmented Generation Systems
por: Liu, Xunzhuo, et al.
Publicado: (2026)

From Inference Routing to Agent Orchestration: Declarative Policy Compilation with Cross-Layer Verification
por: Chen, Huamin, et al.
Publicado: (2026)

inference-fleet-sim: A Queueing-Theory-Grounded Fleet Capacity Planner for LLM Inference
por: Chen, Huamin, et al.
Publicado: (2026)

vLLM Hook v0: A Plug-in for Programming Model Internals on vLLM
por: Ko, Ching-Yun, et al.
Publicado: (2026)

Category-Aware Semantic Caching for Heterogeneous LLM Workloads
por: Wang, Chen, et al.
Publicado: (2025)

vLLM-Omni: Fully Disaggregated Serving for Any-to-Any Multimodal Models
por: Yin, Peiqi, et al.
Publicado: (2026)

Embedding the Teacher: Distilling vLLM Preferences for Scalable Image Retrieval
por: He, Eric, et al.
Publicado: (2025)

Benchmarking Energy Efficiency of Large Language Models Using vLLM
por: Pronk, K., et al.
Publicado: (2025)

History-Aware Cross-Attention Reinforcement: Self-Supervised Multi Turn and Chain-of-Thought Fine-Tuning with vLLM
por: Kiruluta, Andrew, et al.
Publicado: (2025)

R2-Router: A New Paradigm for LLM Routing with Reasoning
por: Xue, Jiaqi, et al.
Publicado: (2026)

LLM Router: Rethinking Routing with Prefill Activations
por: Varshney, Tanay, et al.
Publicado: (2026)

Life-Cycle Routing Vulnerabilities of LLM Router
por: Lin, Qiqi, et al.
Publicado: (2025)

Automated Dynamic AI Inference Scaling on HPC-Infrastructure: Integrating Kubernetes, Slurm and vLLM
por: Trappen, Tim, et al.
Publicado: (2025)

SkillRouter: Skill Routing for LLM Agents at Scale
por: Zheng, YanZhao, et al.
Publicado: (2026)

When Routing Collapses: On the Degenerate Convergence of LLM Routers
por: Lai, Guannan, et al.
Publicado: (2026)

Arch-Router: Aligning LLM Routing with Human Preferences
por: Tran, Co, et al.
Publicado: (2025)

The Impact of Hyperparameters on Large Language Model Inference Performance: An Evaluation of vLLM and HuggingFace Pipelines
por: Martinez, Matias
Publicado: (2024)

RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers
por: Lu, Yifan, et al.
Publicado: (2025)

OmniRouter: Budget and Performance Controllable Multi-LLM Routing
por: Mei, Kai, et al.
Publicado: (2025)

ICL-Router: In-Context Learned Model Representations for LLM Routing
por: Wang, Chenxu, et al.
Publicado: (2025)

RouterBench: A Benchmark for Multi-LLM Routing System
por: Hu, Qitian Jason, et al.
Publicado: (2024)

CP-Router: An Uncertainty-Aware Router Between LLM and LRM
por: Su, Jiayuan, et al.
Publicado: (2025)

How Robust Are Router-LLMs? Analysis of the Fragility of LLM Routing Capabilities
por: Kassem, Aly M., et al.
Publicado: (2025)

Route to Rome Attack: Directing LLM Routers to Expensive Models via Adversarial Suffix Optimization
por: Tang, Haochun, et al.
Publicado: (2026)

ICPC: In-context Prompt Compression with Faster Inference
por: Yu, Ziyang, et al.
Publicado: (2025)

RCR-Router: Efficient Role-Aware Context Routing for Multi-Agent LLM Systems with Structured Memory
por: Liu, Jun, et al.
Publicado: (2025)

Rerouting LLM Routers
por: Shafran, Avital, et al.
Publicado: (2025)

Comparative Analysis of Large Language Model Inference Serving Systems: A Performance Study of vLLM and HuggingFace TGI
por: Kolluru, Saicharan
Publicado: (2025)