98$\times$ Faster LLM Routing Without a Dedicated GPU: Flash Attention, Prompt Compression, and Near-Streaming for the vLLM Semantic Router

Documenti analoghi

• When to Reason: Semantic Router for vLLM
  di: Wang, Chen, et al.
  Pubblicazione: (2025)
• The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Project
  di: Chen, Huamin, et al.
  Pubblicazione: (2026)
• Dual-Pool Token-Budget Routing for Cost-Efficient and Reliable LLM Serving
  di: Liu, Xunzhuo, et al.
  Pubblicazione: (2026)
• Outcome-Aware Tool Selection for Semantic Routers: Latency-Constrained Learning Without LLM Inference
  di: Chen, Huamin, et al.
  Pubblicazione: (2026)
• vLLM Semantic Router: Signal Driven Decision Routing for Mixture-of-Modality Models
  di: Liu, Xunzhuo, et al.
  Pubblicazione: (2026)