Image de couverture de livre

OmniInfer: System-Wide Acceleration Techniques for Optimizing LLM Serving Throughput and Latency

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Wang, Jun, Yao, Yunxiang, Kuang, Wenwei, Mao, Runze, Sun, Zhenhao, Tao, Zhuang, Zhang, Ziyang, Li, Dengyu, Chen, Jiajun, Wang, Zhili, Cui, Kai, Cai, Congzhi, Lan, Longwen, Zhang, Ken
Format:	Preprint
Publié:	2025
Sujets:	Distributed, Parallel, and Cluster Computing
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
par: Agrawal, Amey, et autres
Publié: (2024)

Prompt-Aware Scheduling for Low-Latency LLM Serving
par: Tao, Yiheng, et autres
Publié: (2025)

SSR: Spatial Sequential Hybrid Architecture for Latency Throughput Tradeoff in Transformer Acceleration
par: Zhuang, Jinming, et autres
Publié: (2024)

ADOR: A Design Exploration Framework for LLM Serving with Enhanced Latency and Throughput
par: Kim, Junsoo, et autres
Publié: (2025)

Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Serving
par: Dai, Yinwei, et autres
Publié: (2023)

SEED: Accelerating Reasoning Tree Construction via Scheduled Speculative Decoding
par: Wang, Zhenglin, et autres
Publié: (2024)

CascadeInfer: Length-Aware Scheduling of LLM Serving with Low Latency and Load Balancing
par: Yuan, Yitao, et autres
Publié: (2025)

Vortex: Hosting ML Inference and Knowledge Retrieval Services With Tight Latency and Throughput Requirements
par: Yang, Yuting, et autres
Publié: (2025)

SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification
par: Miao, Xupeng, et autres
Publié: (2023)

Causal Walk: Debiasing Multi-Hop Fact Verification with Front-Door Adjustment
par: Zhang, Congzhi, et autres
Publié: (2024)

Slice-Level Scheduling for High Throughput and Load Balanced LLM Serving
par: Cheng, Ke, et autres
Publié: (2024)

Accelerating Parallel Diffusion Model Serving with Residual Compression
par: Luo, Jiajun, et autres
Publié: (2025)

Two-body interaction induced phase transitions and intermediate phases in nonreciprocal non-Hermitian quasicrystals
par: Zhang, Yalun, et autres
Publié: (2024)

Rethinking Latency Denial-of-Service: Attacking the LLM Serving Framework, Not the Model
par: Wang, Tianyi, et autres
Publié: (2026)

A hybrid reconstruction of piece-wise smooth functions from non-uniform Fourier data
par: Song, Guohui, et autres
Publié: (2026)

FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving
par: Ye, Zihao, et autres
Publié: (2025)

Performance Analysis of uRLLC in scalable Cell-free Radio Access Network System
par: Zhang, Ziyang, et autres
Publié: (2024)

Development and clinical application of a high‐performance medical static computed tomography system
par: Haining Ding, et autres
Publié: (2026)

An End‐to‐End Pillar Feature Based Neural Network Improved by Attention Modules for Object Detection of Autonomous Vehicles
par: Bin Zhang, et autres
Publié: (2025)

Dumbo-NG: Fast Asynchronous BFT Consensus with Throughput-Oblivious Latency
par: Gao, Yingzi, et autres
Publié: (2022)

GRF-based Predictive Flocking Control with Dynamic Pattern Formation
par: Yu, Chenghao, et autres
Publié: (2024)

Quantum geometry and geometric entanglement entropy of one-dimensional Floquet topological matter
par: Zhou, Longwen
Publié: (2024)

Entanglement phase transitions in non-Hermitian Floquet systems
par: Zhou, Longwen
Publié: (2023)

Entanglement phase transitions in non-Hermitian Kitaev chains
par: Zhou, Longwen
Publié: (2024)

Topology and edge modes surviving criticality in non-Hermitian Floquet systems
par: Zhou, Longwen
Publié: (2026)

Non-Abelian generalization of non-Hermitian quasicrystal: PT-symmetry breaking, localization, entanglement and topological transitions
par: Zhou, Longwen
Publié: (2023)

Entanglement phase transitions in non-Hermitian quasicrystals
par: Zhou, Longwen
Publié: (2023)

Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading
par: Yu, Hanfei, et autres
Publié: (2025)

Omni-Judge: Can Omni-LLMs Serve as Human-Aligned Judges for Text-Conditioned Audio-Video Generation?
par: Liang, Susan, et autres
Publié: (2026)

Towards Pareto Optimal Throughput in Small Language Model Serving
par: Recasens, Pol G., et autres
Publié: (2024)

OmniSep: Unified Omni-Modality Sound Separation with Query-Mixup
par: Cheng, Xize, et autres
Publié: (2024)

Falcon: Advancing Asynchronous BFT Consensus for Lower Latency and Enhanced Throughput
par: Dai, Xiaohai, et autres
Publié: (2025)

Size conditions and spectral conditions for generalized factor-critical (bicritical) graphs and $k$-$d$-critical graphs
par: Zhang, Zhenhao, et autres
Publié: (2026)

MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism
par: Zhu, Ruidong, et autres
Publié: (2025)

StreamServe: Adaptive Speculative Flows for Low-Latency Disaggregated LLM Serving
par: Kumar, Satyam, et autres
Publié: (2026)

SegMAN: Omni-scale Context Modeling with State Space Models and Local Attention for Semantic Segmentation
par: Fu, Yunxiang, et autres
Publié: (2024)

PCR: A Prefetch-Enhanced Cache Reuse System for Low-Latency RAG Serving
par: Wang, Wenfeng, et autres
Publié: (2026)

Security, Latency, and Throughput of Proof-of-Work Nakamoto Consensus
par: Cao, Shu-Jie, et autres
Publié: (2023)

NanoFlow: Towards Optimal Large Language Model Serving Throughput
par: Zhu, Kan, et autres
Publié: (2024)

vLLM-Omni: Fully Disaggregated Serving for Any-to-Any Multimodal Models
par: Yin, Peiqi, et autres
Publié: (2026)