Set Block Decoding is a Language Model Inference Accelerator
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909770587308032 |
|---|---|
| author | Gat, Itai Ben-Hamu, Heli Havasi, Marton Haziza, Daniel Reizenstein, Jeremy Synnaeve, Gabriel Lopez-Paz, David Karrer, Brian Lipman, Yaron |
| author_facet | Gat, Itai Ben-Hamu, Heli Havasi, Marton Haziza, Daniel Reizenstein, Jeremy Synnaeve, Gabriel Lopez-Paz, David Karrer, Brian Lipman, Yaron |
| contents | Autoregressive next token prediction language models offer powerful capabilities but face significant challenges in practical deployment due to the high computational and memory costs of inference, particularly during the decoding stage. We introduce Set Block Decoding (SBD), a simple and flexible paradigm that accelerates generation by integrating standard next token prediction (NTP) and masked token prediction (MATP) within a single architecture. SBD allows the model to sample multiple, not necessarily consecutive, future tokens in parallel, a key distinction from previous acceleration methods. This flexibility allows the use of advanced solvers from the discrete diffusion literature, offering significant speedups without sacrificing accuracy. SBD requires no architectural changes or extra training hyperparameters, maintains compatibility with exact KV-caching, and can be implemented by fine-tuning existing next token prediction models. By fine-tuning Llama-3.1 8B and Qwen-3 8B, we demonstrate that SBD enables a 3-5x reduction in the number of forward passes required for generation while achieving same performance as equivalent NTP training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_04185 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Set Block Decoding is a Language Model Inference Accelerator Gat, Itai Ben-Hamu, Heli Havasi, Marton Haziza, Daniel Reizenstein, Jeremy Synnaeve, Gabriel Lopez-Paz, David Karrer, Brian Lipman, Yaron Machine Learning Autoregressive next token prediction language models offer powerful capabilities but face significant challenges in practical deployment due to the high computational and memory costs of inference, particularly during the decoding stage. We introduce Set Block Decoding (SBD), a simple and flexible paradigm that accelerates generation by integrating standard next token prediction (NTP) and masked token prediction (MATP) within a single architecture. SBD allows the model to sample multiple, not necessarily consecutive, future tokens in parallel, a key distinction from previous acceleration methods. This flexibility allows the use of advanced solvers from the discrete diffusion literature, offering significant speedups without sacrificing accuracy. SBD requires no architectural changes or extra training hyperparameters, maintains compatibility with exact KV-caching, and can be implemented by fine-tuning existing next token prediction models. By fine-tuning Llama-3.1 8B and Qwen-3 8B, we demonstrate that SBD enables a 3-5x reduction in the number of forward passes required for generation while achieving same performance as equivalent NTP training. |
| title | Set Block Decoding is a Language Model Inference Accelerator |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2509.04185 |