Copertina

OmniInfer: System-Wide Acceleration Techniques for Optimizing LLM Serving Throughput and Latency

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Wang, Jun, Yao, Yunxiang, Kuang, Wenwei, Mao, Runze, Sun, Zhenhao, Tao, Zhuang, Zhang, Ziyang, Li, Dengyu, Chen, Jiajun, Wang, Zhili, Cui, Kai, Cai, Congzhi, Lan, Longwen, Zhang, Ken
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Distributed, Parallel, and Cluster Computing
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
di: Agrawal, Amey, et al.
Pubblicazione: (2024)

Prompt-Aware Scheduling for Low-Latency LLM Serving
di: Tao, Yiheng, et al.
Pubblicazione: (2025)

SSR: Spatial Sequential Hybrid Architecture for Latency Throughput Tradeoff in Transformer Acceleration
di: Zhuang, Jinming, et al.
Pubblicazione: (2024)

ADOR: A Design Exploration Framework for LLM Serving with Enhanced Latency and Throughput
di: Kim, Junsoo, et al.
Pubblicazione: (2025)

Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Serving
di: Dai, Yinwei, et al.
Pubblicazione: (2023)

SEED: Accelerating Reasoning Tree Construction via Scheduled Speculative Decoding
di: Wang, Zhenglin, et al.
Pubblicazione: (2024)

CascadeInfer: Length-Aware Scheduling of LLM Serving with Low Latency and Load Balancing
di: Yuan, Yitao, et al.
Pubblicazione: (2025)

Vortex: Hosting ML Inference and Knowledge Retrieval Services With Tight Latency and Throughput Requirements
di: Yang, Yuting, et al.
Pubblicazione: (2025)

SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification
di: Miao, Xupeng, et al.
Pubblicazione: (2023)

Causal Walk: Debiasing Multi-Hop Fact Verification with Front-Door Adjustment
di: Zhang, Congzhi, et al.
Pubblicazione: (2024)

Slice-Level Scheduling for High Throughput and Load Balanced LLM Serving
di: Cheng, Ke, et al.
Pubblicazione: (2024)

Accelerating Parallel Diffusion Model Serving with Residual Compression
di: Luo, Jiajun, et al.
Pubblicazione: (2025)

Two-body interaction induced phase transitions and intermediate phases in nonreciprocal non-Hermitian quasicrystals
di: Zhang, Yalun, et al.
Pubblicazione: (2024)

Rethinking Latency Denial-of-Service: Attacking the LLM Serving Framework, Not the Model
di: Wang, Tianyi, et al.
Pubblicazione: (2026)

A hybrid reconstruction of piece-wise smooth functions from non-uniform Fourier data
di: Song, Guohui, et al.
Pubblicazione: (2026)

FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving
di: Ye, Zihao, et al.
Pubblicazione: (2025)

Performance Analysis of uRLLC in scalable Cell-free Radio Access Network System
di: Zhang, Ziyang, et al.
Pubblicazione: (2024)

Development and clinical application of a high‐performance medical static computed tomography system
di: Haining Ding, et al.
Pubblicazione: (2026)

Dumbo-NG: Fast Asynchronous BFT Consensus with Throughput-Oblivious Latency
di: Gao, Yingzi, et al.
Pubblicazione: (2022)

An End‐to‐End Pillar Feature Based Neural Network Improved by Attention Modules for Object Detection of Autonomous Vehicles
di: Bin Zhang, et al.
Pubblicazione: (2025)

Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading
di: Yu, Hanfei, et al.
Pubblicazione: (2025)

GRF-based Predictive Flocking Control with Dynamic Pattern Formation
di: Yu, Chenghao, et al.
Pubblicazione: (2024)

Quantum geometry and geometric entanglement entropy of one-dimensional Floquet topological matter
di: Zhou, Longwen
Pubblicazione: (2024)

Entanglement phase transitions in non-Hermitian Floquet systems
di: Zhou, Longwen
Pubblicazione: (2023)

Entanglement phase transitions in non-Hermitian Kitaev chains
di: Zhou, Longwen
Pubblicazione: (2024)

Topology and edge modes surviving criticality in non-Hermitian Floquet systems
di: Zhou, Longwen
Pubblicazione: (2026)

Non-Abelian generalization of non-Hermitian quasicrystal: PT-symmetry breaking, localization, entanglement and topological transitions
di: Zhou, Longwen
Pubblicazione: (2023)

Entanglement phase transitions in non-Hermitian quasicrystals
di: Zhou, Longwen
Pubblicazione: (2023)

Omni-Judge: Can Omni-LLMs Serve as Human-Aligned Judges for Text-Conditioned Audio-Video Generation?
di: Liang, Susan, et al.
Pubblicazione: (2026)

Towards Pareto Optimal Throughput in Small Language Model Serving
di: Recasens, Pol G., et al.
Pubblicazione: (2024)

OmniSep: Unified Omni-Modality Sound Separation with Query-Mixup
di: Cheng, Xize, et al.
Pubblicazione: (2024)

Falcon: Advancing Asynchronous BFT Consensus for Lower Latency and Enhanced Throughput
di: Dai, Xiaohai, et al.
Pubblicazione: (2025)

Size conditions and spectral conditions for generalized factor-critical (bicritical) graphs and $k$-$d$-critical graphs
di: Zhang, Zhenhao, et al.
Pubblicazione: (2026)

MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism
di: Zhu, Ruidong, et al.
Pubblicazione: (2025)

StreamServe: Adaptive Speculative Flows for Low-Latency Disaggregated LLM Serving
di: Kumar, Satyam, et al.
Pubblicazione: (2026)

SegMAN: Omni-scale Context Modeling with State Space Models and Local Attention for Semantic Segmentation
di: Fu, Yunxiang, et al.
Pubblicazione: (2024)

PCR: A Prefetch-Enhanced Cache Reuse System for Low-Latency RAG Serving
di: Wang, Wenfeng, et al.
Pubblicazione: (2026)

Security, Latency, and Throughput of Proof-of-Work Nakamoto Consensus
di: Cao, Shu-Jie, et al.
Pubblicazione: (2023)

NanoFlow: Towards Optimal Large Language Model Serving Throughput
di: Zhu, Kan, et al.
Pubblicazione: (2024)

vLLM-Omni: Fully Disaggregated Serving for Any-to-Any Multimodal Models
di: Yin, Peiqi, et al.
Pubblicazione: (2026)