AIBrix: Towards Scalable, Cost-Effective Large Language Model Inference Infrastructure
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915228836429824 |
|---|---|
| author | The AIBrix Team Shan, Jiaxin Gupta, Varun Xu, Le Shi, Haiyang Zhang, Jingyuan Wang, Ning Xu, Linhui Kang, Rong Liu, Tongping Zhang, Yifei Zhu, Yiqing Jin, Shuowei Lim, Gangmuk Chen, Binbin Chen, Zuzhi Liu, Xiao Chen, Xin Yin, Kante Chung, Chak-Pong Jiang, Chenyu Lu, Yicheng Chen, Jianjun Lin, Caixue Xiang, Wu Shi, Rui Xie, Liguang |
| author_facet | The AIBrix Team Shan, Jiaxin Gupta, Varun Xu, Le Shi, Haiyang Zhang, Jingyuan Wang, Ning Xu, Linhui Kang, Rong Liu, Tongping Zhang, Yifei Zhu, Yiqing Jin, Shuowei Lim, Gangmuk Chen, Binbin Chen, Zuzhi Liu, Xiao Chen, Xin Yin, Kante Chung, Chak-Pong Jiang, Chenyu Lu, Yicheng Chen, Jianjun Lin, Caixue Xiang, Wu Shi, Rui Xie, Liguang |
| contents | We introduce AIBrix, a cloud-native, open-source framework designed to optimize and simplify large-scale LLM deployment in cloud environments. Unlike traditional cloud-native stacks, AIBrix follows a co-design philosophy, ensuring every layer of the infrastructure is purpose-built for seamless integration with inference engines like vLLM. AIBrix introduces several key innovations to reduce inference costs and enhance performance including high-density LoRA management for dynamic adapter scheduling, LLM-specific autoscalers, and prefix-aware, load-aware routing. To further improve efficiency, AIBrix incorporates a distributed KV cache, boosting token reuse across nodes, leading to a 50% increase in throughput and a 70% reduction in inference latency. AIBrix also supports unified AI runtime which streamlines model management while maintaining vendor-agnostic engine compatibility. For large-scale multi-node inference, AIBrix employs hybrid orchestration -- leveraging Kubernetes for coarse-grained scheduling and Ray for fine-grained execution -- to balance efficiency and flexibility. Additionally, an SLO-driven GPU optimizer dynamically adjusts resource allocations, optimizing heterogeneous serving to maximize cost efficiency while maintaining service guarantees. Finally, AIBrix enhances system reliability with AI accelerator diagnostic tools, enabling automated failure detection and mock-up testing to improve fault resilience. AIBrix is available at https://github.com/vllm-project/aibrix. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_03648 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AIBrix: Towards Scalable, Cost-Effective Large Language Model Inference Infrastructure The AIBrix Team Shan, Jiaxin Gupta, Varun Xu, Le Shi, Haiyang Zhang, Jingyuan Wang, Ning Xu, Linhui Kang, Rong Liu, Tongping Zhang, Yifei Zhu, Yiqing Jin, Shuowei Lim, Gangmuk Chen, Binbin Chen, Zuzhi Liu, Xiao Chen, Xin Yin, Kante Chung, Chak-Pong Jiang, Chenyu Lu, Yicheng Chen, Jianjun Lin, Caixue Xiang, Wu Shi, Rui Xie, Liguang Distributed, Parallel, and Cluster Computing Artificial Intelligence We introduce AIBrix, a cloud-native, open-source framework designed to optimize and simplify large-scale LLM deployment in cloud environments. Unlike traditional cloud-native stacks, AIBrix follows a co-design philosophy, ensuring every layer of the infrastructure is purpose-built for seamless integration with inference engines like vLLM. AIBrix introduces several key innovations to reduce inference costs and enhance performance including high-density LoRA management for dynamic adapter scheduling, LLM-specific autoscalers, and prefix-aware, load-aware routing. To further improve efficiency, AIBrix incorporates a distributed KV cache, boosting token reuse across nodes, leading to a 50% increase in throughput and a 70% reduction in inference latency. AIBrix also supports unified AI runtime which streamlines model management while maintaining vendor-agnostic engine compatibility. For large-scale multi-node inference, AIBrix employs hybrid orchestration -- leveraging Kubernetes for coarse-grained scheduling and Ray for fine-grained execution -- to balance efficiency and flexibility. Additionally, an SLO-driven GPU optimizer dynamically adjusts resource allocations, optimizing heterogeneous serving to maximize cost efficiency while maintaining service guarantees. Finally, AIBrix enhances system reliability with AI accelerator diagnostic tools, enabling automated failure detection and mock-up testing to improve fault resilience. AIBrix is available at https://github.com/vllm-project/aibrix. |
| title | AIBrix: Towards Scalable, Cost-Effective Large Language Model Inference Infrastructure |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence |
| url | https://arxiv.org/abs/2504.03648 |