Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915926166732800 |
|---|---|
| author | Siavashi, Mohammad Scazzariello, Mariano Maguire Jr., Gerald Q. Kostić, Dejan Chiesa, Marco |
| author_facet | Siavashi, Mohammad Scazzariello, Mariano Maguire Jr., Gerald Q. Kostić, Dejan Chiesa, Marco |
| contents | Large Language Model (LLM) inference is rapidly becoming a core datacenter service, yet current serving stacks keep the host CPU on the critical path for orchestration and token-level control. This makes LLM performance sensitive to CPU interference, undermining application colocation and forcing operators to reserve CPU headroom, leaving substantial capacity unutilized.
We introduce Blink, an end-to-end serving architecture that removes the host CPU from the steady-state inference path by redistributing responsibilities across a SmartNIC and a GPU. Blink offloads request handling to the SmartNIC, which delivers inputs directly into GPU memory via RDMA, and replaces host-driven scheduling with a persistent GPU kernel that performs batching, scheduling, and KV-cache management without CPU involvement.
Evaluated against TensorRT-LLM, vLLM, and SGLang, Blink outperforms all baselines even in isolation, reducing pre-saturation P99 TTFT by up to 8.47$\times$ and P99 TPOT by up to 3.40$\times$, improving decode throughput by up to 2.1$\times$, and reducing energy per token by up to 48.6$\%$. Under CPU interference, Blink maintains stable performance, while existing systems degrade by up to two orders of magnitude. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_07609 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC Siavashi, Mohammad Scazzariello, Mariano Maguire Jr., Gerald Q. Kostić, Dejan Chiesa, Marco Distributed, Parallel, and Cluster Computing Machine Learning Operating Systems Performance Software Engineering Large Language Model (LLM) inference is rapidly becoming a core datacenter service, yet current serving stacks keep the host CPU on the critical path for orchestration and token-level control. This makes LLM performance sensitive to CPU interference, undermining application colocation and forcing operators to reserve CPU headroom, leaving substantial capacity unutilized. We introduce Blink, an end-to-end serving architecture that removes the host CPU from the steady-state inference path by redistributing responsibilities across a SmartNIC and a GPU. Blink offloads request handling to the SmartNIC, which delivers inputs directly into GPU memory via RDMA, and replaces host-driven scheduling with a persistent GPU kernel that performs batching, scheduling, and KV-cache management without CPU involvement. Evaluated against TensorRT-LLM, vLLM, and SGLang, Blink outperforms all baselines even in isolation, reducing pre-saturation P99 TTFT by up to 8.47$\times$ and P99 TPOT by up to 3.40$\times$, improving decode throughput by up to 2.1$\times$, and reducing energy per token by up to 48.6$\%$. Under CPU interference, Blink maintains stable performance, while existing systems degrade by up to two orders of magnitude. |
| title | Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC |
| topic | Distributed, Parallel, and Cluster Computing Machine Learning Operating Systems Performance Software Engineering |
| url | https://arxiv.org/abs/2604.07609 |