QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917923385245696 |
|---|---|
| author | Tiwari, Rishabh Xi, Haocheng Tomar, Aditya Hooper, Coleman Kim, Sehoon Horton, Maxwell Najibi, Mahyar Mahoney, Michael W. Keutzer, Kurt Gholami, Amir |
| author_facet | Tiwari, Rishabh Xi, Haocheng Tomar, Aditya Hooper, Coleman Kim, Sehoon Horton, Maxwell Najibi, Mahyar Mahoney, Michael W. Keutzer, Kurt Gholami, Amir |
| contents | Large Language Models (LLMs) are increasingly being deployed on edge devices for long-context settings, creating a growing need for fast and efficient long-context inference. In these scenarios, the Key-Value (KV) cache is the primary bottleneck in terms of both GPU memory and latency, as the full KV cache must be loaded for each decoding step. While speculative decoding is a widely accepted technique to accelerate autoregressive decoding, existing methods often struggle to achieve significant speedups due to inefficient KV cache optimization strategies and result in low acceptance rates. To address these challenges, we propose a novel self-speculative decoding framework, QuantSpec, where the draft model shares the architecture of the target model but employs a hierarchical 4-bit quantized KV cache and 4-bit quantized weights for acceleration. QuantSpec maintains high acceptance rates ($>$90%) and reliably provides consistent end-to-end speedups upto $\sim2.5\times$, outperforming other self-speculative decoding methods that use sparse KV cache for long-context LLM inference. QuantSpec also reduces the memory requirements by $\sim 1.3\times$ compared to these alternatives. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_10424 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache Tiwari, Rishabh Xi, Haocheng Tomar, Aditya Hooper, Coleman Kim, Sehoon Horton, Maxwell Najibi, Mahyar Mahoney, Michael W. Keutzer, Kurt Gholami, Amir Machine Learning Artificial Intelligence Large Language Models (LLMs) are increasingly being deployed on edge devices for long-context settings, creating a growing need for fast and efficient long-context inference. In these scenarios, the Key-Value (KV) cache is the primary bottleneck in terms of both GPU memory and latency, as the full KV cache must be loaded for each decoding step. While speculative decoding is a widely accepted technique to accelerate autoregressive decoding, existing methods often struggle to achieve significant speedups due to inefficient KV cache optimization strategies and result in low acceptance rates. To address these challenges, we propose a novel self-speculative decoding framework, QuantSpec, where the draft model shares the architecture of the target model but employs a hierarchical 4-bit quantized KV cache and 4-bit quantized weights for acceleration. QuantSpec maintains high acceptance rates ($>$90%) and reliably provides consistent end-to-end speedups upto $\sim2.5\times$, outperforming other self-speculative decoding methods that use sparse KV cache for long-context LLM inference. QuantSpec also reduces the memory requirements by $\sim 1.3\times$ compared to these alternatives. |
| title | QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2502.10424 |