Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.10576 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915669232058368 |
|---|---|
| author | Chen, Xinhang Zhang, Chao He, Jiahuan Liu, Wei Zhang, Jianming Zhou, Wenlong Li, Xiao Zeng, Pai Li, Shiyong Qian, Yuanpan Li, Dong Li, Zhaogeng |
| author_facet | Chen, Xinhang Zhang, Chao He, Jiahuan Liu, Wei Zhang, Jianming Zhou, Wenlong Li, Xiao Zeng, Pai Li, Shiyong Qian, Yuanpan Li, Dong Li, Zhaogeng |
| contents | DeepSeek-V3.2-Exp introduces a sparse attention mechanism that significantly reduces inference latency in long-context scenarios. Although the overall throughput has improved greatly, the Decode-stage of PD disaggregation remains to be a major bottleneck. This bottleneck primarily stems from the conflict between linear growth of Latent-Cache with sequence length and the limited GPU memory capacity, which constrains the feasible batch-size and thereby suppresses Decode-stage throughput.
To address this challenge, we propose ESS (Extended Sparse Server), an offload-centric system design tailored for DeepSeek-V3.2-Exp. ESS selectively offloads Latent-Cache to CPU memory while preserving latency-critical components on GPU. By freeing up GPU memory, ESS effectively decoupling batch-size scaling from GPU memory constraints. This design significantly improves Decode-stage throughput, thereby reducing deployment costs in real-world settings.
Our high-fidelity simulations show that ESS delivers 69.4\% throughput improvement at 32K context length and up to 123\% throughput improvement at 128K, demonstrating its effectiveness for large-context inference workloads. These results highlight ESS as a practical and scalable solution for long-context LLM serving. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_10576 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ESS: An Offload-Centric Latent-Cache Management Architecture for DeepSeek-V3.2-Exp Chen, Xinhang Zhang, Chao He, Jiahuan Liu, Wei Zhang, Jianming Zhou, Wenlong Li, Xiao Zeng, Pai Li, Shiyong Qian, Yuanpan Li, Dong Li, Zhaogeng Distributed, Parallel, and Cluster Computing DeepSeek-V3.2-Exp introduces a sparse attention mechanism that significantly reduces inference latency in long-context scenarios. Although the overall throughput has improved greatly, the Decode-stage of PD disaggregation remains to be a major bottleneck. This bottleneck primarily stems from the conflict between linear growth of Latent-Cache with sequence length and the limited GPU memory capacity, which constrains the feasible batch-size and thereby suppresses Decode-stage throughput. To address this challenge, we propose ESS (Extended Sparse Server), an offload-centric system design tailored for DeepSeek-V3.2-Exp. ESS selectively offloads Latent-Cache to CPU memory while preserving latency-critical components on GPU. By freeing up GPU memory, ESS effectively decoupling batch-size scaling from GPU memory constraints. This design significantly improves Decode-stage throughput, thereby reducing deployment costs in real-world settings. Our high-fidelity simulations show that ESS delivers 69.4\% throughput improvement at 32K context length and up to 123\% throughput improvement at 128K, demonstrating its effectiveness for large-context inference workloads. These results highlight ESS as a practical and scalable solution for long-context LLM serving. |
| title | ESS: An Offload-Centric Latent-Cache Management Architecture for DeepSeek-V3.2-Exp |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2512.10576 |