Cover Image

Intelligent Router for LLM Workloads: Improving Performance Through Workload-Aware Load Balancing

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Jain, Kunal, Parayil, Anjaly, Mallick, Ankur, Choukse, Esha, Qin, Xiaoting, Zhang, Jue, Goiri, Íñigo, Wang, Rujia, Bansal, Chetan, Rühle, Victor, Kulkarni, Anoop, Kofsky, Steve, Rajmohan, Saravan
Format:	Preprint
Published:	2024
Subjects:	Distributed, Parallel, and Cluster Computing Systems and Control
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

SageServe: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling
by: Jaiswal, Shashwat, et al.
Published: (2025)

Towards Cloud Efficiency with Large-scale Workload Characterization
by: Parayil, Anjaly, et al.
Published: (2024)

Ensuring Fair LLM Serving Amid Diverse Applications
by: Khan, Redwan Ibne Seraj, et al.
Published: (2024)

Workload Intelligence: Punching Holes Through the Cloud Abstraction
by: Huang, Lexiang, et al.
Published: (2024)

Attention Enhanced Entity Recommendation for Intelligent Monitoring in Cloud Systems
by: Hussain, Fiza, et al.
Published: (2025)

Intelligent Monitoring Framework for Cloud Services: A Data-Driven Approach
by: Srinivas, Pooja, et al.
Published: (2024)

A Holistic Framework for Automated Configuration Recommendation for Cloud Service Monitoring
by: Bastos, Anson, et al.
Published: (2026)

X-lifecycle Learning for Cloud Incident Management using LLMs
by: Goel, Drishti, et al.
Published: (2024)

DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency
by: Stojkovic, Jovan, et al.
Published: (2024)

Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inference
by: Stojkovic, Jovan, et al.
Published: (2024)

Memora: A Harmonic Memory Representation Balancing Abstraction and Specificity
by: Xia, Menglin, et al.
Published: (2026)

Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systems
by: Jaiswal, Shashwat, et al.
Published: (2025)

Cost-Aware Retrieval-Augmentation Reasoning Models with Adaptive Retrieval Depth
by: Hashemi, Helia, et al.
Published: (2025)

Automated Root Causing of Cloud Incidents using In-Context Learning with GPT-4
by: Zhang, Xuchao, et al.
Published: (2024)

Towards Resource-Efficient Compound AI Systems
by: Chaudhry, Gohar Irfan, et al.
Published: (2025)

Splitwise: Efficient generative LLM inference using phase splitting
by: Patel, Pratyush, et al.
Published: (2023)

Risk-aware Adaptive Virtual CPU Oversubscription in Microsoft Cloud via Prototypical Human-in-the-loop Imitation Learning
by: Wang, Lu, et al.
Published: (2024)

Murakkab: Resource-Efficient Agentic Workflow Orchestration in Cloud Platforms
by: Chaudhry, Gohar Irfan, et al.
Published: (2025)

StreamWise: Serving Multi-Modal Generation in Real-Time at Scale
by: Qiu, Haoran, et al.
Published: (2026)

ModServe: Modality- and Stage-Aware Resource Disaggregation for Scalable Multimodal Model Serving
by: Qiu, Haoran, et al.
Published: (2025)

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection
by: Gupta, Taneesh, et al.
Published: (2025)

REFA: Reference Free Alignment for multi-preference optimization
by: Gupta, Taneesh, et al.
Published: (2024)

TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platforms
by: Stojkovic, Jovan, et al.
Published: (2025)

Exploring How LLMs Capture and Represent Domain-Specific Knowledge
by: Garcia, Mirian Hipolito, et al.
Published: (2025)

AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Clouds
by: Chen, Yinfang, et al.
Published: (2025)

Junctiond: Extending FaaS Runtimes with Kernel-Bypass
by: Saurez, Enrique, et al.
Published: (2024)

Sutradhara: An Intelligent Orchestrator-Engine Co-design for Tool-based Agentic Inference
by: Biswas, Anish, et al.
Published: (2026)

Revisiting Transformer Layer Parameterization Through Causal Energy Minimization
by: Xu, Jin, et al.
Published: (2026)

Semantic Caching of Contextual Summaries for Efficient Question-Answering with Language Models
by: Couturier, Camille, et al.
Published: (2025)

Sherlock: Reliable and Efficient Agentic Workflow Execution
by: Ro, Yeonju, et al.
Published: (2025)

No Request Left Behind: Tackling Heterogeneity in Long-Context LLM Inference with Medha
by: Agrawal, Amey, et al.
Published: (2024)

AutoAdapt: An Automated Domain Adaptation Framework for LLMs
by: Sinha, Sidharth, et al.
Published: (2026)

Multi-Preference Optimization: Generalizing DPO via Set-Level Contrasts
by: Gupta, Taneesh, et al.
Published: (2024)

Synergistic Weak-Strong Collaboration by Aligning Preferences
by: Jiao, Yizhu, et al.
Published: (2025)

Lean Attention: Hardware-Aware Scalable Attention Mechanism for the Decode-Phase of Transformers
by: Sanovar, Rya, et al.
Published: (2024)

Load Balancing for AI Training Workloads
by: McClure, Sarah, et al.
Published: (2025)

From Reasoning to Answer: Empirical, Attention-Based and Mechanistic Insights into Distilled DeepSeek R1 Models
by: Zhang, Jue, et al.
Published: (2025)

An Advanced Reinforcement Learning Framework for Online Scheduling of Deferrable Workloads in Cloud Computing
by: Dong, Hang, et al.
Published: (2024)

Input-Dependent Power Usage in GPUs
by: Gregersen, Theo, et al.
Published: (2024)

TurboAttention: Efficient Attention Approximation For High Throughputs LLMs
by: Kang, Hao, et al.
Published: (2024)