Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systems
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911292013412352 |
|---|---|
| author | Jaiswal, Shashwat Arun, Shrikara Parayil, Anjaly Mallick, Ankur Mastorakis, Spyros Khare, Alind Alverti, Chloi Amant, Renee St Bansal, Chetan Rühle, Victor Torrellas, Josep |
| author_facet | Jaiswal, Shashwat Arun, Shrikara Parayil, Anjaly Mallick, Ankur Mastorakis, Spyros Khare, Alind Alverti, Chloi Amant, Renee St Bansal, Chetan Rühle, Victor Torrellas, Josep |
| contents | Low-Rank Adaptation (LoRA) has become the de facto method for parameter-efficient fine-tuning of large language models (LLMs), enabling rapid adaptation to diverse domains. In production, LoRA-based models are served at scale, creating multi-tenant environments with hundreds of adapters sharing a base model. However, state-of-the-art serving systems co-batch heterogeneous adapters without accounting for rank (size) variability, leading to severe performance skew, which ultimately requires adding more GPUs to satisfy service-level objectives (SLOs). Existing optimizations, focused on loading, caching, and kernel execution, ignore this heterogeneity, leaving GPU resources underutilized. We present LoRAServe, a workload-aware dynamic adapter placement and routing framework designed to tame rank diversity in LoRA serving. By dynamically rebalancing adapters across GPUs and leveraging GPU Direct RDMA for remote access, LoRAServe maximizes throughput and minimizes tail latency under real-world workload drift. Evaluations on production traces from Company X show that LoRAServe elicits up to 2$\times$ higher throughput, up to 9$\times$ lower TTFT, while using up to 50% fewer GPUs under SLO constraints compared to state-of-the-art systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_22880 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systems Jaiswal, Shashwat Arun, Shrikara Parayil, Anjaly Mallick, Ankur Mastorakis, Spyros Khare, Alind Alverti, Chloi Amant, Renee St Bansal, Chetan Rühle, Victor Torrellas, Josep Distributed, Parallel, and Cluster Computing Artificial Intelligence Machine Learning Low-Rank Adaptation (LoRA) has become the de facto method for parameter-efficient fine-tuning of large language models (LLMs), enabling rapid adaptation to diverse domains. In production, LoRA-based models are served at scale, creating multi-tenant environments with hundreds of adapters sharing a base model. However, state-of-the-art serving systems co-batch heterogeneous adapters without accounting for rank (size) variability, leading to severe performance skew, which ultimately requires adding more GPUs to satisfy service-level objectives (SLOs). Existing optimizations, focused on loading, caching, and kernel execution, ignore this heterogeneity, leaving GPU resources underutilized. We present LoRAServe, a workload-aware dynamic adapter placement and routing framework designed to tame rank diversity in LoRA serving. By dynamically rebalancing adapters across GPUs and leveraging GPU Direct RDMA for remote access, LoRAServe maximizes throughput and minimizes tail latency under real-world workload drift. Evaluations on production traces from Company X show that LoRAServe elicits up to 2$\times$ higher throughput, up to 9$\times$ lower TTFT, while using up to 50% fewer GPUs under SLO constraints compared to state-of-the-art systems. |
| title | Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systems |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2511.22880 |