Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2601.06288 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915719769227264 |
|---|---|
| author | Xu, Tianhao Liu, Yiming Lu, Xianglong Zhao, Yijia Zhou, Xuting Feng, Aichen Chen, Yiyi Shen, Yi Zhou, Qin Chen, Xumeng Sherstyuk, Ilya Li, Haorui Thakkar, Rishi Hamm, Ben Li, Yuanzhe Huang, Xue Wu, Wenpeng Shanbhag, Anish Kim, Harry Chen, Chuan Lai, Junjie |
| author_facet | Xu, Tianhao Liu, Yiming Lu, Xianglong Zhao, Yijia Zhou, Xuting Feng, Aichen Chen, Yiyi Shen, Yi Zhou, Qin Chen, Xumeng Sherstyuk, Ilya Li, Haorui Thakkar, Rishi Hamm, Ben Li, Yuanzhe Huang, Xue Wu, Wenpeng Shanbhag, Anish Kim, Harry Chen, Chuan Lai, Junjie |
| contents | Optimizing Large Language Model (LLM) inference in production systems is increasingly difficult due to dynamic workloads, stringent latency/throughput targets, and a rapidly expanding configuration space. This complexity spans not only distributed parallelism strategies (tensor/pipeline/expert) but also intricate framework-specific runtime parameters such as those concerning the enablement of CUDA graphs, available KV-cache memory fractions, and maximum token capacity, which drastically impact performance. The diversity of modern inference frameworks (e.g., TRT-LLM, vLLM, SGLang), each employing distinct kernels and execution policies, makes manual tuning both framework-specific and computationally prohibitive. We present AIConfigurator, a unified performance-modeling system that enables rapid, framework-agnostic inference configuration search without requiring GPU-based profiling. AIConfigurator combines (1) a methodology that decomposes inference into analytically modelable primitives - GEMM, attention, communication, and memory operations while capturing framework-specific scheduling dynamics; (2) a calibrated kernel-level performance database for these primitives across a wide range of hardware platforms and popular open-weights models (GPT-OSS, Qwen, DeepSeek, LLama, Mistral); and (3) an abstraction layer that automatically resolves optimal launch parameters for the target backend, seamlessly integrating into production-grade orchestration systems. Evaluation on production LLM serving workloads demonstrates that AIConfigurator identifies superior serving configurations that improve performance by up to 40% for dense models (e.g., Qwen3-32B) and 50% for MoE architectures (e.g., DeepSeek-V3), while completing searches within 30 seconds on average. Enabling the rapid exploration of vast design spaces - from cluster topology down to engine specific flags. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_06288 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving Xu, Tianhao Liu, Yiming Lu, Xianglong Zhao, Yijia Zhou, Xuting Feng, Aichen Chen, Yiyi Shen, Yi Zhou, Qin Chen, Xumeng Sherstyuk, Ilya Li, Haorui Thakkar, Rishi Hamm, Ben Li, Yuanzhe Huang, Xue Wu, Wenpeng Shanbhag, Anish Kim, Harry Chen, Chuan Lai, Junjie Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing Optimizing Large Language Model (LLM) inference in production systems is increasingly difficult due to dynamic workloads, stringent latency/throughput targets, and a rapidly expanding configuration space. This complexity spans not only distributed parallelism strategies (tensor/pipeline/expert) but also intricate framework-specific runtime parameters such as those concerning the enablement of CUDA graphs, available KV-cache memory fractions, and maximum token capacity, which drastically impact performance. The diversity of modern inference frameworks (e.g., TRT-LLM, vLLM, SGLang), each employing distinct kernels and execution policies, makes manual tuning both framework-specific and computationally prohibitive. We present AIConfigurator, a unified performance-modeling system that enables rapid, framework-agnostic inference configuration search without requiring GPU-based profiling. AIConfigurator combines (1) a methodology that decomposes inference into analytically modelable primitives - GEMM, attention, communication, and memory operations while capturing framework-specific scheduling dynamics; (2) a calibrated kernel-level performance database for these primitives across a wide range of hardware platforms and popular open-weights models (GPT-OSS, Qwen, DeepSeek, LLama, Mistral); and (3) an abstraction layer that automatically resolves optimal launch parameters for the target backend, seamlessly integrating into production-grade orchestration systems. Evaluation on production LLM serving workloads demonstrates that AIConfigurator identifies superior serving configurations that improve performance by up to 40% for dense models (e.g., Qwen3-32B) and 50% for MoE architectures (e.g., DeepSeek-V3), while completing searches within 30 seconds on average. Enabling the rapid exploration of vast design spaces - from cluster topology down to engine specific flags. |
| title | AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving |
| topic | Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2601.06288 |