The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Project

Documenti analoghi

• Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inference
  di: Chen, Huamin, et al.
  Pubblicazione: (2026)
• FleetOpt: Analytical Fleet Provisioning for LLM Inference with Compress-and-Route as Implementation Mechanism
  di: Chen, Huamin, et al.
  Pubblicazione: (2026)
• inference-fleet-sim: A Queueing-Theory-Grounded Fleet Capacity Planner for LLM Inference
  di: Chen, Huamin, et al.
  Pubblicazione: (2026)
• The 1/W Law: An Analytical Study of Context-Length Routing Topology and GPU Generation Gains for LLM Inference Energy Efficiency
  di: Chen, Huamin, et al.
  Pubblicazione: (2026)
• When to Reason: Semantic Router for vLLM
  di: Wang, Chen, et al.
  Pubblicazione: (2025)