Distributed Speculative Inference (DSI): Speculation Parallelism for Provably Faster Lossless Language Model Inference
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910876264562688 |
|---|---|
| author | Timor, Nadav Mamou, Jonathan Korat, Daniel Berchansky, Moshe Pereg, Oren Wasserblat, Moshe Galanti, Tomer Gordon, Michal Harel, David |
| author_facet | Timor, Nadav Mamou, Jonathan Korat, Daniel Berchansky, Moshe Pereg, Oren Wasserblat, Moshe Galanti, Tomer Gordon, Michal Harel, David |
| contents | This paper introduces distributed speculative inference (DSI), a novel inference algorithm that is provably faster than speculative inference (SI) [leviathan2023, chen2023, miao2024, sun2025, timor2025] and standard autoregressive inference (non-SI). Like other SI algorithms, DSI operates on frozen language models (LMs), requiring no training or architectural modifications, and it preserves the target distribution. Prior studies on SI have demonstrated empirical speedups over non-SI--but rely on sufficiently fast and accurate drafters, which are often unavailable in practice. We identify a gap where SI can be slower than non-SI if drafters are too slow or inaccurate. We close this gap by proving that DSI is faster than both SI and non-SI--given any drafters. DSI is therefore not only faster than SI, but also unlocks the acceleration of LMs for which SI fails. DSI leverages speculation parallelism (SP), a novel type of task parallelism, to orchestrate target and drafter instances that overlap in time, establishing a new foundational tradeoff between computational resources and latency. Our simulations show that DSI is 1.29-1.92x faster than SI in single-node setups for various off-the-shelf LMs and tasks. We open-source all our code. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_14105 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Distributed Speculative Inference (DSI): Speculation Parallelism for Provably Faster Lossless Language Model Inference Timor, Nadav Mamou, Jonathan Korat, Daniel Berchansky, Moshe Pereg, Oren Wasserblat, Moshe Galanti, Tomer Gordon, Michal Harel, David Distributed, Parallel, and Cluster Computing Artificial Intelligence Computation and Language Machine Learning This paper introduces distributed speculative inference (DSI), a novel inference algorithm that is provably faster than speculative inference (SI) [leviathan2023, chen2023, miao2024, sun2025, timor2025] and standard autoregressive inference (non-SI). Like other SI algorithms, DSI operates on frozen language models (LMs), requiring no training or architectural modifications, and it preserves the target distribution. Prior studies on SI have demonstrated empirical speedups over non-SI--but rely on sufficiently fast and accurate drafters, which are often unavailable in practice. We identify a gap where SI can be slower than non-SI if drafters are too slow or inaccurate. We close this gap by proving that DSI is faster than both SI and non-SI--given any drafters. DSI is therefore not only faster than SI, but also unlocks the acceleration of LMs for which SI fails. DSI leverages speculation parallelism (SP), a novel type of task parallelism, to orchestrate target and drafter instances that overlap in time, establishing a new foundational tradeoff between computational resources and latency. Our simulations show that DSI is 1.29-1.92x faster than SI in single-node setups for various off-the-shelf LMs and tasks. We open-source all our code. |
| title | Distributed Speculative Inference (DSI): Speculation Parallelism for Provably Faster Lossless Language Model Inference |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2405.14105 |