Arctic Inference with Shift Parallelism: Fast and Efficient Open Source Inference System for Enterprise AI
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913943561175040 |
|---|---|
| author | Rajbhandari, Samyam Hidayetoglu, Mert Qiao, Aurick Wang, Ye Yang, Juncheng Rasley, Jeff Wyatt, Michael He, Yuxiong |
| author_facet | Rajbhandari, Samyam Hidayetoglu, Mert Qiao, Aurick Wang, Ye Yang, Juncheng Rasley, Jeff Wyatt, Michael He, Yuxiong |
| contents | Inference is now the dominant AI workload, yet existing systems force trade-offs between latency, throughput, and cost. Arctic Inference, an open-source vLLM plugin from Snowflake AI Research, introduces Shift Parallelism, a dynamic parallelism strategy that adapts to real-world traffic while integrating speculative decoding, SwiftKV compute reduction, and optimized embedding inference. It achieves up to 3.4 times faster request completion, 1.75 times faster generation, and 1.6M tokens/sec per GPU for embeddings, outperforming both latency- and throughput-optimized deployments. Already powering Snowflake Cortex AI, Arctic Inference delivers state-of-the-art, cost-effective inference for enterprise AI and is now available to the community. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_11830 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Arctic Inference with Shift Parallelism: Fast and Efficient Open Source Inference System for Enterprise AI Rajbhandari, Samyam Hidayetoglu, Mert Qiao, Aurick Wang, Ye Yang, Juncheng Rasley, Jeff Wyatt, Michael He, Yuxiong Distributed, Parallel, and Cluster Computing Machine Learning Inference is now the dominant AI workload, yet existing systems force trade-offs between latency, throughput, and cost. Arctic Inference, an open-source vLLM plugin from Snowflake AI Research, introduces Shift Parallelism, a dynamic parallelism strategy that adapts to real-world traffic while integrating speculative decoding, SwiftKV compute reduction, and optimized embedding inference. It achieves up to 3.4 times faster request completion, 1.75 times faster generation, and 1.6M tokens/sec per GPU for embeddings, outperforming both latency- and throughput-optimized deployments. Already powering Snowflake Cortex AI, Arctic Inference delivers state-of-the-art, cost-effective inference for enterprise AI and is now available to the community. |
| title | Arctic Inference with Shift Parallelism: Fast and Efficient Open Source Inference System for Enterprise AI |
| topic | Distributed, Parallel, and Cluster Computing Machine Learning |
| url | https://arxiv.org/abs/2507.11830 |