Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.16273 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914247095615488 |
|---|---|
| author | Zheng, Ce Zhang, Ke Sun, Chen Zhang, Wenqi Liu, Qiong Tesfay, Angesom Ataklity |
| author_facet | Zheng, Ce Zhang, Ke Sun, Chen Zhang, Wenqi Liu, Qiong Tesfay, Angesom Ataklity |
| contents | Speculative decoding accelerates large language model (LLM) inference by allowing a small draft model to predict multiple future tokens for verification by a larger target model. In AI-native radio access networks (AI-RAN), this enables device-edge collaborative inference but introduces significant uplink overhead, as existing distributed speculative decoding schemes transmit full vocabulary logits at every step. We propose a sparsify-then-sample strategy, Truncated Sparse Logits Transmission (TSLT), which transmits only the logits and indices of a truncated candidate set. We provide theoretical guarantees showing that the acceptance rate is preserved under TSLT. TSLT is further extended to multi-candidate case, where multiple draft candidates per step increase acceptance probability. Experiments show that TSLT significantly reduces uplink communication while maintaining end-to-end inference latency and model quality, demonstrating its effectiveness for scalable, communication-efficient distributed LLM inference in future AI-RAN systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_16273 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Fast Collaborative Inference via Distributed Speculative Decoding Zheng, Ce Zhang, Ke Sun, Chen Zhang, Wenqi Liu, Qiong Tesfay, Angesom Ataklity Signal Processing Speculative decoding accelerates large language model (LLM) inference by allowing a small draft model to predict multiple future tokens for verification by a larger target model. In AI-native radio access networks (AI-RAN), this enables device-edge collaborative inference but introduces significant uplink overhead, as existing distributed speculative decoding schemes transmit full vocabulary logits at every step. We propose a sparsify-then-sample strategy, Truncated Sparse Logits Transmission (TSLT), which transmits only the logits and indices of a truncated candidate set. We provide theoretical guarantees showing that the acceptance rate is preserved under TSLT. TSLT is further extended to multi-candidate case, where multiple draft candidates per step increase acceptance probability. Experiments show that TSLT significantly reduces uplink communication while maintaining end-to-end inference latency and model quality, demonstrating its effectiveness for scalable, communication-efficient distributed LLM inference in future AI-RAN systems. |
| title | Fast Collaborative Inference via Distributed Speculative Decoding |
| topic | Signal Processing |
| url | https://arxiv.org/abs/2512.16273 |