Image de couverture de livre

NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Jiang, Xuanlin, Zhou, Yang, Cao, Shiyi, Stoica, Ion, Yu, Minlan
Format:	Preprint
Publié:	2024
Sujets:	Distributed, Parallel, and Cluster Computing Artificial Intelligence Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Pie: Pooling CPU Memory for LLM Inference
par: Xu, Yi, et autres
Publié: (2024)

MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs
par: Cao, Shiyi, et autres
Publié: (2024)

Prism: Unleashing GPU Sharing for Cost-Efficient Multi-LLM Serving
par: Yu, Shan, et autres
Publié: (2025)

TriMoE: Augmenting GPU with AMX-Enabled CPU and DIMM-NDP for High-Throughput MoE Inference via Offloading
par: Pan, Yudong, et autres
Publié: (2026)

UCCL-Zip: Lossless Compression Supercharged GPU Communication
par: Ma, Shuang, et autres
Publié: (2026)

Characterizing and Optimizing LLM Inference Workloads on CPU-GPU Coupled Architectures
par: Vellaisamy, Prabhu, et autres
Publié: (2025)

T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge
par: Wei, Jianyu, et autres
Publié: (2024)

Breaking the Memory Wall: A Study of I/O Patterns and GPU Memory Utilization for Hybrid CPU-GPU Offloaded Optimizers
par: Maurya, Avinash, et autres
Publié: (2024)

PALS: Power-Aware LLM Serving for Mixture-of-Experts Models
par: Hankendi, Can, et autres
Publié: (2026)

APEX: Asynchronous Parallel CPU-GPU Execution for Online LLM Inference on Constrained GPUs
par: Fan, Jiakun, et autres
Publié: (2025)

TURNIP: A "Nondeterministic" GPU Runtime with CPU RAM Offload
par: Ding, Zhimin, et autres
Publié: (2024)

DAK: Direct-Access-Enabled GPU Memory Offloading with Optimal Efficiency for LLM Inference
par: Lin, Shouxu, et autres
Publié: (2026)

A Parallel CPU-GPU Framework for Batching Heuristic Operations in Depth-First Heuristic Search
par: Futuhi, Ehsan, et autres
Publié: (2025)

DUAL-BLADE: Dual-Path NVMe-Direct KV-Cache Offloading for Edge LLM Inference
par: Jeong, Bodon, et autres
Publié: (2026)

SiPipe: Bridging the CPU-GPU Utilization Gap for Efficient Pipeline-Parallel LLM Inference
par: He, Yongchao, et autres
Publié: (2025)

Investigating Memory Failure Prediction Across CPU Architectures
par: Yu, Qiao, et autres
Publié: (2024)

Taming GPU Underutilization via Static Partitioning and Fine-grained CPU Offloading
par: Schieffer, Gabin, et autres
Publié: (2026)

ODIN-Based CPU-GPU Architecture with Replay-Driven Simulation and Emulation
par: Dorairaj, Nij, et autres
Publié: (2026)

MSCCL++: Rethinking GPU Communication Abstractions for AI Inference
par: Hwang, Changho, et autres
Publié: (2025)

Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures
par: Argerich, Mauricio Fadel, et autres
Publié: (2026)

A Few GPUs, A Whole Lotta Scale: Faithful LLM Training Emulation with PrismLLM
par: Xi, Shaoke, et autres
Publié: (2026)

Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models
par: Kamahori, Keisuke, et autres
Publié: (2024)

KVSwap: Disk-aware KV Cache Offloading for Long-Context On-device Inference
par: Zhang, Huawei, et autres
Publié: (2025)

FairKV: Balancing Per-Head KV Cache for Fast Multi-GPU Inference
par: Zhao, Bingzhe, et autres
Publié: (2025)

Efficient CPU-GPU Collaborative Inference for MoE-based LLMs on Memory-Limited Systems
par: Huang, En-Ming, et autres
Publié: (2025)

Llamas on the Web: Memory-Efficient, Performance-Portable, and Multi-Precision LLM Inference with WebGPU
par: Levine, Reese, et autres
Publié: (2026)

Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption
par: Yildiz, Mert, et autres
Publié: (2026)

Cost-Efficient Multimodal LLM Inference via Cross-Tier GPU Heterogeneity
par: Yu, Donglin
Publié: (2026)

Power- and Fragmentation-aware Online Scheduling for GPU Datacenters
par: Lettich, Francesco, et autres
Publié: (2024)

Taming Asynchronous CPU-GPU Coupling for Frequency-aware Latency Estimation on Mobile Edge
par: Chen, Jiesong, et autres
Publié: (2026)

Characterizing CPU-Induced Slowdowns in Multi-GPU LLM Inference
par: Chung, Euijun, et autres
Publié: (2026)

SkyServe: Serving AI Models across Regions and Clouds with Spot Instances
par: Mao, Ziming, et autres
Publié: (2024)

LLM Inference Serving: Survey of Recent Advances and Opportunities
par: Li, Baolin, et autres
Publié: (2024)

Accurate GPU Memory Prediction for Deep Learning Jobs through Dynamic Analysis
par: Shi, Jiabo, et autres
Publié: (2025)

Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading
par: Yu, Hanfei, et autres
Publié: (2025)

DISTFLASHATTN: Distributed Memory-efficient Attention for Long-context LLMs Training
par: Li, Dacheng, et autres
Publié: (2023)

S-LoRA: Serving Thousands of Concurrent LoRA Adapters
par: Sheng, Ying, et autres
Publié: (2023)

ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Serving
par: Qiao, Yifan, et autres
Publié: (2024)

HybridGen: Efficient LLM Generative Inference via CPU-GPU Hybrid Computing
par: Lin, Mao, et autres
Publié: (2026)

PackKV: Reducing KV Cache Memory Footprint through LLM-Aware Lossy Compression
par: Jiang, Bo, et autres
Publié: (2025)