AnchorTP: Resilient LLM Inference with State-Preserving Elastic Tensor Parallelism
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912710105497600 |
|---|---|
| author | Xu, Wendong Chen, Chujie Xiao, He Li, Kuan Xiong, Jing Zhang, Chen Zhou, Wenyong Tao, Chaofan Bai, Yang Yu, Bei Wong, Ngai |
| author_facet | Xu, Wendong Chen, Chujie Xiao, He Li, Kuan Xiong, Jing Zhang, Chen Zhou, Wenyong Tao, Chaofan Bai, Yang Yu, Bei Wong, Ngai |
| contents | Large Language Model (LLM) inference services demand exceptionally high availability and low latency, yet multi-GPU Tensor Parallelism (TP) makes them vulnerable to single-GPU failures. We present AnchorTP, a state-preserving elastic TP framework for fast recovery. It (i) enables Elastic Tensor Parallelism (ETP) with unequal-width partitioning over any number of GPUs and compatibility with Mixture-of-Experts (MoE), and (ii) preserves model parameters and KV caches in GPU memory via a daemon decoupled from the inference process. To minimize downtime, we propose a bandwidth-aware planner based on a Continuous Minimal Migration (CMM) algorithm that minimizes reload bytes under a byte-cost dominance assumption, and an execution scheduler that pipelines P2P transfers with reloads. These components jointly restore service quickly with minimal data movement and without changing service interfaces. In typical failure scenarios, AnchorTP reduces Time to First Success (TFS) by up to 11x and Time to Peak (TTP) by up to 59% versus restart-and-reload. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_11617 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AnchorTP: Resilient LLM Inference with State-Preserving Elastic Tensor Parallelism Xu, Wendong Chen, Chujie Xiao, He Li, Kuan Xiong, Jing Zhang, Chen Zhou, Wenyong Tao, Chaofan Bai, Yang Yu, Bei Wong, Ngai Distributed, Parallel, and Cluster Computing Large Language Model (LLM) inference services demand exceptionally high availability and low latency, yet multi-GPU Tensor Parallelism (TP) makes them vulnerable to single-GPU failures. We present AnchorTP, a state-preserving elastic TP framework for fast recovery. It (i) enables Elastic Tensor Parallelism (ETP) with unequal-width partitioning over any number of GPUs and compatibility with Mixture-of-Experts (MoE), and (ii) preserves model parameters and KV caches in GPU memory via a daemon decoupled from the inference process. To minimize downtime, we propose a bandwidth-aware planner based on a Continuous Minimal Migration (CMM) algorithm that minimizes reload bytes under a byte-cost dominance assumption, and an execution scheduler that pipelines P2P transfers with reloads. These components jointly restore service quickly with minimal data movement and without changing service interfaces. In typical failure scenarios, AnchorTP reduces Time to First Success (TFS) by up to 11x and Time to Peak (TTP) by up to 59% versus restart-and-reload. |
| title | AnchorTP: Resilient LLM Inference with State-Preserving Elastic Tensor Parallelism |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2511.11617 |