Image de couverture de livre

Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Agrawal, Amey, Kedia, Nitin, Panwar, Ashish, Mohan, Jayashree, Kwatra, Nipun, Gulavani, Bhargav S., Tumanov, Alexey, Ramjee, Ramachandran
Format:	Preprint
Publié:	2024
Sujets:	Machine Learning Distributed, Parallel, and Cluster Computing
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

On Evaluating Performance of LLM Inference Serving Systems
par: Agrawal, Amey, et autres
Publié: (2025)

Etalon: Holistic Performance Evaluation Framework for LLM Inference Systems
par: Agrawal, Amey, et autres
Publié: (2024)

Vidur: A Large-Scale Simulation Framework For LLM Inference
par: Agrawal, Amey, et autres
Publié: (2024)

Niyama : Breaking the Silos of LLM Inference Serving
par: Goel, Kanishk, et autres
Publié: (2025)

TokenWeave: Efficient Compute-Communication Overlap for Distributed LLM Inference
par: Gond, Raja, et autres
Publié: (2025)

Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference
par: Deshmukh, Dhruv, et autres
Publié: (2025)

POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference
par: Kamath, Aditya K, et autres
Publié: (2024)

No Request Left Behind: Tackling Heterogeneity in Long-Context LLM Inference with Medha
par: Agrawal, Amey, et autres
Publié: (2024)

LLM-42: Enabling Determinism in LLM Inference with Verified Speculation
par: Gond, Raja, et autres
Publié: (2026)

Sutradhara: An Intelligent Orchestrator-Engine Co-design for Tool-based Agentic Inference
par: Biswas, Anish, et autres
Publié: (2026)

Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Serving
par: Dai, Yinwei, et autres
Publié: (2023)

ModServe: Modality- and Stage-Aware Resource Disaggregation for Scalable Multimodal Model Serving
par: Qiu, Haoran, et autres
Publié: (2025)

Revati: Transparent GPU-Free Time-Warp Emulation for LLM Serving
par: Agrawal, Amey, et autres
Publié: (2026)

OmniInfer: System-Wide Acceleration Techniques for Optimizing LLM Serving Throughput and Latency
par: Wang, Jun, et autres
Publié: (2025)

SuperFedNAS: Cost-Efficient Federated Neural Architecture Search for On-Device Inference
par: Khare, Alind, et autres
Publié: (2023)

Exploration of Energy and Throughput Tradeoffs for Dataflow Networks
par: Karim, Abrarul, et autres
Publié: (2026)

Shift Parallelism: Low-Latency, High-Throughput LLM Inference for Dynamic Workloads
par: Hidayetoglu, Mert, et autres
Publié: (2025)

Maya: Optimizing Deep Learning Training Workloads using GPU Runtime Emulation
par: Yarlagadda, Srihas, et autres
Publié: (2025)

HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds
par: Lou, Chiheng, et autres
Publié: (2025)

Speculative Decoding in Decentralized LLM Inference: Turning Communication Latency into Computation Throughput
par: Song, Jingwei, et autres
Publié: (2025)

Falcon: Advancing Asynchronous BFT Consensus for Lower Latency and Enhanced Throughput
par: Dai, Xiaohai, et autres
Publié: (2025)

Taming the Memory Footprint Crisis: System Design for Production Diffusion LLM Serving
par: Fan, Jiakun, et autres
Publié: (2025)

NanoFlow: Towards Optimal Large Language Model Serving Throughput
par: Zhu, Kan, et autres
Publié: (2024)

Slice-Level Scheduling for High Throughput and Load Balanced LLM Serving
par: Cheng, Ke, et autres
Publié: (2024)

BBCA-CHAIN: Low Latency, High Throughput BFT Consensus on a DAG
par: Malkhi, Dahlia, et autres
Publié: (2023)

OOCO: Latency-disaggregated Architecture for Online-Offline Co-locate LLM Serving
par: Wu, Siyu, et autres
Publié: (2025)

StreamServe: Adaptive Speculative Flows for Low-Latency Disaggregated LLM Serving
par: Kumar, Satyam, et autres
Publié: (2026)

PCR: A Prefetch-Enhanced Cache Reuse System for Low-Latency RAG Serving
par: Wang, Wenfeng, et autres
Publié: (2026)

CascadeInfer: Length-Aware Scheduling of LLM Serving with Low Latency and Load Balancing
par: Yuan, Yitao, et autres
Publié: (2025)

Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading
par: Yu, Hanfei, et autres
Publié: (2025)

Taming the Titans: A Survey of Efficient LLM Inference Serving
par: Zhen, Ranran, et autres
Publié: (2025)

Argus: Quality-Aware High-Throughput Text-to-Image Inference Serving System
par: Agarwal, Shubham, et autres
Publié: (2025)

Efficient Graph-Based Approximate Nearest Neighbor Search Achieving: Low Latency Without Throughput Loss
par: Luo, Jingjia, et autres
Publié: (2025)

Cloud Native System for LLM Inference Serving
par: Xu, Minxian, et autres
Publié: (2025)

Serving Compound Inference Systems on Datacenter GPUs
par: Devata, Sriram, et autres
Publié: (2026)

CaraServe: CPU-Assisted and Rank-Aware LoRA Serving for Generative LLM Inference
par: Li, Suyi, et autres
Publié: (2024)

GoodServe: Towards High-Goodput Serving of Agentic LLM Inferences over Heterogeneous Resources
par: Du, Boxiao, et autres
Publié: (2026)

Taming Request Imbalance: SLO-Aware Scheduling for Disaggregated LLM Inference
par: Wang, Qipeng
Publié: (2026)

vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention
par: Prabhu, Ramya, et autres
Publié: (2024)

BrownoutServe: SLO-Aware Inference Serving under Bursty Workloads for MoE-based LLMs
par: Hu, Jianmin, et autres
Publié: (2025)