CascadeInfer: Length-Aware Scheduling of LLM Serving with Low Latency and Load Balancing

Documenti analoghi

• SIMPLE: Disaggregating Sampling from GPU Inference into a Decision Plane for Faster Distributed LLM Serving
  di: Zhao, Bohan, et al.
  Pubblicazione: (2025)
• Fine-grained MoE Load Balancing with Linear Programming
  di: Zhao, Chenqi, et al.
  Pubblicazione: (2025)
• SiPipe: Bridging the CPU-GPU Utilization Gap for Efficient Pipeline-Parallel LLM Inference
  di: He, Yongchao, et al.
  Pubblicazione: (2025)
• Slice-Level Scheduling for High Throughput and Load Balanced LLM Serving
  di: Cheng, Ke, et al.
  Pubblicazione: (2024)
• Prompt-Aware Scheduling for Low-Latency LLM Serving
  di: Tao, Yiheng, et al.
  Pubblicazione: (2025)