Copertina

Memory Offloading for Large Language Model Inference with Latency SLO Guarantees

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Ma, Chenxiang, Ye, Zhisheng, Zhao, Hanyu, Yang, Zehua, Fu, Tianhao, Han, Jiaxun, Zhang, Jie, Luo, Yingwei, Wang, Xiaolin, Wang, Zhenlin, Li, Yong, Zhou, Diyu
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Distributed, Parallel, and Cluster Computing
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

SLO-Aware Task Offloading within Collaborative Vehicle Platoons
di: Sedlak, Boris, et al.
Pubblicazione: (2024)

LatencyPrism: Online Non-intrusive Latency Sculpting for SLO-Guaranteed LLM Inference
di: Du, Yin, et al.
Pubblicazione: (2026)

Joint Optimization of Offloading, Batching and DVFS for Multiuser Co-Inference
di: Xu, Yaodan, et al.
Pubblicazione: (2025)

A Heuristic Algorithm for Shortest Path Search
di: Yu, Huashan, et al.
Pubblicazione: (2025)

Accelerating Mixture-of-Experts Inference by Hiding Offloading Latency with Speculative Decoding
di: Wang, Zhibin, et al.
Pubblicazione: (2025)

FlexKV: Flexible Index Offloading for Memory-Disaggregated Key-Value Store
di: Hu, Zhisheng, et al.
Pubblicazione: (2025)

Taming Request Imbalance: SLO-Aware Scheduling for Disaggregated LLM Inference
di: Wang, Qipeng
Pubblicazione: (2026)

SCOOT: SLO-Oriented Performance Tuning for LLM Inference Engines
di: Cheng, Ke, et al.
Pubblicazione: (2024)

EconoServe: Maximizing Multi-Resource Utilization with SLO Guarantees in LLM Serving
di: Shen, Haiying, et al.
Pubblicazione: (2024)

SLO-Aware Scheduling for Large Language Model Inferences
di: Huang, Jinqi, et al.
Pubblicazione: (2025)

Serving Hybrid LLM Loads with SLO Guarantees Using CPU-GPU Attention Piggybacking
di: Mo, Zizhao, et al.
Pubblicazione: (2026)

BrownoutServe: SLO-Aware Inference Serving under Bursty Workloads for MoE-based LLMs
di: Hu, Jianmin, et al.
Pubblicazione: (2025)

Orchestrating Joint Offloading and Scheduling for Low-Latency Edge SLAM
di: Zhang, Yao, et al.
Pubblicazione: (2025)

SLICE: SLO-Driven Scheduling for LLM Inference on Edge Computing Devices
di: Chow, Will
Pubblicazione: (2025)

Breaking the Memory Wall: A Study of I/O Patterns and GPU Memory Utilization for Hybrid CPU-GPU Offloaded Optimizers
di: Maurya, Avinash, et al.
Pubblicazione: (2024)

HarmonyBatch: Batching multi-SLO DNN Inference with Heterogeneous Serverless Functions
di: Chen, Jiabin, et al.
Pubblicazione: (2024)

A House United Within Itself: SLO-Awareness for On-Premises Containerized ML Inference Clusters via Faro
di: Jeon, Beomyeol, et al.
Pubblicazione: (2024)

DAK: Direct-Access-Enabled GPU Memory Offloading with Optimal Efficiency for LLM Inference
di: Lin, Shouxu, et al.
Pubblicazione: (2026)

MemAscend: System Memory Optimization for SSD-Offloaded LLM Fine-Tuning
di: Liaw, Yong-Cheng, et al.
Pubblicazione: (2025)

HAS-GPU: Efficient Hybrid Auto-scaling with Fine-grained GPU Allocation for SLO-aware Serverless Inferences
di: Gu, Jianfeng, et al.
Pubblicazione: (2025)

Workload Distribution with Rateless Encoding: A Low-Latency Computation Offloading Method within Edge Networks
di: Guo, Zhongfu, et al.
Pubblicazione: (2023)

FlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guarantees
di: Oliaro, Gabriele, et al.
Pubblicazione: (2024)

Torpor: GPU-Enabled Serverless Computing for Low-Latency, Resource-Efficient Inference
di: Yu, Minchen, et al.
Pubblicazione: (2023)

Hummingbird: SLO-Oriented GPU Preemption at Microsecond-scale
di: Hu, Tiancheng, et al.
Pubblicazione: (2026)

Characterization of Large Language Model Development in the Datacenter
di: Hu, Qinghao, et al.
Pubblicazione: (2024)

Automatic BLAS Offloading on Unified Memory Architecture: A Study on NVIDIA Grace-Hopper
di: Li, Junjie, et al.
Pubblicazione: (2024)

LSRAM: A Lightweight Autoscaling and SLO Resource Allocation Framework for Microservices Based on Gradient Descent
di: Hu, Kan, et al.
Pubblicazione: (2024)

PATCHEDSERVE: A Patch Management Framework for SLO-Optimized Hybrid Resolution Diffusion Serving
di: Sun, Desen, et al.
Pubblicazione: (2025)

MSARS: A Meta-Learning and Reinforcement Learning Framework for SLO Resource Allocation and Adaptive Scaling for Microservices
di: Hu, Kan, et al.
Pubblicazione: (2024)

Collaborative Inference for Large Models with Task Offloading and Early Exiting
di: Xie, Zuan, et al.
Pubblicazione: (2024)

Aladdin: Joint Placement and Scaling for SLO-Aware LLM Serving
di: Nie, Chengyi, et al.
Pubblicazione: (2024)

MaaSO: SLO-aware Orchestration of Heterogeneous Model Instances for MaaS
di: Xuan, Mo, et al.
Pubblicazione: (2025)

PromptTuner: SLO-Aware Elastic System for LLM Prompt Tuning
di: Gao, Wei, et al.
Pubblicazione: (2026)

AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains
di: Kumar, Abhishek Vijaya, et al.
Pubblicazione: (2024)

Disaggregated Memory with SmartNIC Offloading: a Case Study on Graph Processing
di: Wahlgren, Jacob, et al.
Pubblicazione: (2024)

LA-IMR: Latency-Aware, Predictive In-Memory Routing and Proactive Autoscaling for Tail-Latency-Sensitive Cloud Robotics
di: Seo, Eunil, et al.
Pubblicazione: (2025)

Accelerating Edge Inference for Distributed MoE Models with Latency-Optimized Expert Placement
di: Wu, Tian, et al.
Pubblicazione: (2025)

CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Control
di: Chen, Qiaoling, et al.
Pubblicazione: (2026)

SuperInfer: SLO-Aware Rotary Scheduling and Memory Management for LLM Inference on Superchips
di: Yu, Jiahuan, et al.
Pubblicazione: (2026)

Tangram: High-resolution Video Analytics on Serverless Platform with SLO-aware Batching
di: Peng, Haosong, et al.
Pubblicazione: (2024)