Copertina

Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Recasens, Pol G., Agullo, Ferran, Zhu, Yue, Wang, Chen, Lee, Eun Kyung, Tardieu, Olivier, Torres, Jordi, Berral, Josep Ll.
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Distributed, Parallel, and Cluster Computing Machine Learning
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Data Driven Optimization of GPU efficiency for Distributed LLM Adapter Serving
di: Agullo, Ferran, et al.
Pubblicazione: (2026)

Enabling an OpenStack-based cloud on top of RISC-V hardware
di: Marrón, Diego, et al.
Pubblicazione: (2024)

Optimizing LLM Inference Throughput via Memory-aware and SLA-constrained Dynamic Batching
di: Pang, Bowen, et al.
Pubblicazione: (2025)

SiPipe: Bridging the CPU-GPU Utilization Gap for Efficient Pipeline-Parallel LLM Inference
di: He, Yongchao, et al.
Pubblicazione: (2025)

DAK: Direct-Access-Enabled GPU Memory Offloading with Optimal Efficiency for LLM Inference
di: Lin, Shouxu, et al.
Pubblicazione: (2026)

GPU-Accelerated Batch-Dynamic Subgraph Matching
di: Qiu, Linshan, et al.
Pubblicazione: (2024)

DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference
di: Wu, Yongtong, et al.
Pubblicazione: (2026)

FairBatching: Fairness-Aware Batch Formation for LLM Inference
di: Lyu, Hongtao, et al.
Pubblicazione: (2025)

PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers
di: Zhang, Hongbin, et al.
Pubblicazione: (2026)

HarmonyBatch: Batching multi-SLO DNN Inference with Heterogeneous Serverless Functions
di: Chen, Jiabin, et al.
Pubblicazione: (2024)

CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Control
di: Chen, Qiaoling, et al.
Pubblicazione: (2026)

Efficient CPU-GPU Collaborative Inference for MoE-based LLMs on Memory-Limited Systems
di: Huang, En-Ming, et al.
Pubblicazione: (2025)

Frenzy: A Memory-Aware Serverless LLM Training System for Heterogeneous GPU Clusters
di: Chang, Zihan, et al.
Pubblicazione: (2024)

APEX: Asynchronous Parallel CPU-GPU Execution for Online LLM Inference on Constrained GPUs
di: Fan, Jiakun, et al.
Pubblicazione: (2025)

SYMPHONY: Improving Memory Management for LLM Inference Workloads
di: Agarwal, Saurabh, et al.
Pubblicazione: (2024)

MultiPath Memory Access: Breaking Host-GPU Bandwidth Bottlenecks in LLM Services
di: Tang, Lingfeng, et al.
Pubblicazione: (2025)

Understanding the Landscape of Ampere GPU Memory Errors
di: Zhu, Zhu, et al.
Pubblicazione: (2025)

Breaking the Memory Wall: A Study of I/O Patterns and GPU Memory Utilization for Hybrid CPU-GPU Offloaded Optimizers
di: Maurya, Avinash, et al.
Pubblicazione: (2024)

Joint Optimization of Offloading, Batching and DVFS for Multiuser Co-Inference
di: Xu, Yaodan, et al.
Pubblicazione: (2025)

LMDeploy Accelerates Mixed-Precision LLM Inference with TurboMind
di: Zhang, Li, et al.
Pubblicazione: (2025)

SIMPLE: Disaggregating Sampling from GPU Inference into a Decision Plane for Faster Distributed LLM Serving
di: Zhao, Bohan, et al.
Pubblicazione: (2025)

Towards Efficient Key-Value Cache Management for Prefix Prefilling in LLM Inference
di: Zhu, Yue, et al.
Pubblicazione: (2025)

SiDP: Memory-Efficient Data Parallelism for Offline LLM Inference
di: Zhao, Alan, et al.
Pubblicazione: (2026)

ParvaGPU: Efficient Spatial GPU Sharing for Large-Scale DNN Inference in Cloud Environments
di: Lee, Munkyu, et al.
Pubblicazione: (2024)

Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading
di: Meng, William, et al.
Pubblicazione: (2025)

Scaling Up Throughput-oriented LLM Inference Applications on Heterogeneous Opportunistic GPU Clusters with Pervasive Context Management
di: Phung, Thanh Son, et al.
Pubblicazione: (2025)

Byzantine-Tolerant Consensus in GPU-Inspired Shared Memory
di: Georgiou, Chryssis, et al.
Pubblicazione: (2025)

Predictable LLM Serving on GPU Clusters
di: Darzi, Erfan, et al.
Pubblicazione: (2025)

LIME:Accelerating Collaborative Lossless LLM Inference on Memory-Constrained Edge Devices
di: Sun, Mingyu, et al.
Pubblicazione: (2025)

Efficiently Executing High-throughput Lightweight LLM Inference Applications on Heterogeneous Opportunistic GPU Clusters with Pervasive Context Management
di: Phung, Thanh Son, et al.
Pubblicazione: (2025)

MoE-Gen: High-Throughput MoE Inference on a Single GPU with Module-Based Batching
di: Xu, Tairan, et al.
Pubblicazione: (2025)

PilotANN: Memory-Bounded GPU Acceleration for Vector Search
di: Gui, Yuntao, et al.
Pubblicazione: (2025)

HAS-GPU: Efficient Hybrid Auto-scaling with Fine-grained GPU Allocation for SLO-aware Serverless Inferences
di: Gu, Jianfeng, et al.
Pubblicazione: (2025)

The 1/W Law: An Analytical Study of Context-Length Routing Topology and GPU Generation Gains for LLM Inference Energy Efficiency
di: Chen, Huamin, et al.
Pubblicazione: (2026)

HybridGen: Efficient LLM Generative Inference via CPU-GPU Hybrid Computing
di: Lin, Mao, et al.
Pubblicazione: (2026)

GMLake: Efficient and Transparent GPU Memory Defragmentation for Large-scale DNN Training with Virtual Memory Stitching
di: Guo, Cong, et al.
Pubblicazione: (2024)

AntBatchInfer: Elastic Batch Inference in the Kubernetes Cluster
di: Li, Siyuan, et al.
Pubblicazione: (2024)

Characterizing CPU-Induced Slowdowns in Multi-GPU LLM Inference
di: Chung, Euijun, et al.
Pubblicazione: (2026)

PackInfer: Compute- and I/O-Efficient Attention for Batched LLM Inference
di: Ning, Rui, et al.
Pubblicazione: (2026)

GPU-Accelerated Vecchia Approximations of Gaussian Processes for Geospatial Data using Batched Matrix Computations
di: Pan, Qilong, et al.
Pubblicazione: (2024)