Conformal Sparsification for Bandwidth-Efficient Edge-Cloud Speculative Decoding
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909837277790208 |
|---|---|
| author | Bhattacharjee, Payel Tian, Fengwei Zhong, Meiyu Zhang, Guangyi Simeone, Osvaldo Tandon, Ravi |
| author_facet | Bhattacharjee, Payel Tian, Fengwei Zhong, Meiyu Zhang, Guangyi Simeone, Osvaldo Tandon, Ravi |
| contents | Edge-cloud speculative decoding (SD) accelerates inference by having a cloud-based large language model (LLM) that verifies draft tokens generated by a resource-constrained small language model (SLM) at the edge. A central bottleneck is the limited bandwidth of the edge-cloud link, which necessitates efficient compression of draft token distributions. We first derive an information-theoretic bound that decomposes the token rejection rate into contributions from SLM-LLM distribution mismatch and from quantization distortion. Guided by this analysis, we propose the Sparse Quantize-and-Sample SD (SQS-SD) framework, which exploits distributional sparsity through structured sparsification and lattice-based quantization. Within this framework, K-SQS applies fixed top-K truncation, while C-SQS adaptively adjusts the retained token set via online conformal prediction to ensure bounded deviation from the dense distribution. Empirical results confirm that both approaches improve end-to-end latency and rejection rates in complimentary operating regimes. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_09942 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Conformal Sparsification for Bandwidth-Efficient Edge-Cloud Speculative Decoding Bhattacharjee, Payel Tian, Fengwei Zhong, Meiyu Zhang, Guangyi Simeone, Osvaldo Tandon, Ravi Machine Learning Artificial Intelligence Information Theory Edge-cloud speculative decoding (SD) accelerates inference by having a cloud-based large language model (LLM) that verifies draft tokens generated by a resource-constrained small language model (SLM) at the edge. A central bottleneck is the limited bandwidth of the edge-cloud link, which necessitates efficient compression of draft token distributions. We first derive an information-theoretic bound that decomposes the token rejection rate into contributions from SLM-LLM distribution mismatch and from quantization distortion. Guided by this analysis, we propose the Sparse Quantize-and-Sample SD (SQS-SD) framework, which exploits distributional sparsity through structured sparsification and lattice-based quantization. Within this framework, K-SQS applies fixed top-K truncation, while C-SQS adaptively adjusts the retained token set via online conformal prediction to ensure bounded deviation from the dense distribution. Empirical results confirm that both approaches improve end-to-end latency and rejection rates in complimentary operating regimes. |
| title | Conformal Sparsification for Bandwidth-Efficient Edge-Cloud Speculative Decoding |
| topic | Machine Learning Artificial Intelligence Information Theory |
| url | https://arxiv.org/abs/2510.09942 |