TurboQuant on Quantized Models: Solving Compound Quantization Error with Pre-RoPE Compression and KV Compaction
Fuente:
Zenodo
Enregistré dans:
| Auteur principal: | |
|---|---|
| Format: | Recurso digital |
| Langue: | anglais |
| Publié: |
Zenodo
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866901465632604160 |
|---|---|
| author | Gökyıldız, Onur |
| author_facet | Gökyıldız, Onur |
| contents | <p>Every TurboQuant implementation we tested produces catastrophic output on GGUF quantized models. The reason is compound quantization error: weight quantization noise gets amplified through a second round of KV cache quantization, and softmax makes it worse. On Qwen 2.5 7B Q4_K_M, a representative Python implementation produces perplexity of 3,556 (baseline: 5.18). That is not a typo.</p> <p>This paper presents tq-kv, a Rust implementation of TurboQuant (ICLR 2026) that actually works on quantized models. We solve compound error through the 3-Fix Framework (sink token preservation, past-only quantizatioKV cache compressionn, cache state management) and introduce Pre-RoPE key quantization, which compresses keys before rotary position embedding to exploit position-independent channel statistics. Pre-RoPE reduces the perplexity gap by 34-59% at identical compression ratio. We also integrate KV Compaction, an attention-matching token reduction method orthogonal to bit compression, achieving combined compression up to 100-400x.</p> <p>On the systems side, we replace the dense QJL projection from the original paper with a structured Hadamard variant (SRHT-QJL) that is 115x faster and produces +4.5 dB better SNR. Fused attention scoring eliminates key decompression via AVX2+FMA centroid lookup, yielding 6-8.9x speedup. An O(1) incremental cache architecture cuts per-token overhead from 608 ms to 0.65 ms.</p> <p>We evaluate across five architectures (Llama-3, Qwen2.5, Gemma, Mistral, Phi-3) on both FP16 and GGUF Q4_K_M models. At 4-bit, tq-kv achieves +3.7% PPL delta with Pre-RoPE on GGUF (vs +17% for standard TQ and 3,556 for existing implementations), 9/9 needle-in-a-haystack pass rate, and 7.5-14.2x key compression. The system is 13.7K lines of Rust, ships as a crate on crates.io, and includes a complete inference engine with OpenAI-compatible API.</p> |
| format | Recurso digital |
| id | zenodo_https___doi_org_10_5281_zenodo_19375408 |
| institution | Zenodo |
| language | eng |
| publishDate | 2026 |
| publisher | Zenodo |
| record_format | zenodo |
| spellingShingle | TurboQuant on Quantized Models: Solving Compound Quantization Error with Pre-RoPE Compression and KV Compaction Gökyıldız, Onur KV cache compression compound quantization error Pre-RoPE quantization KV compaction TurboQuant Rust GGUF Lloyd-Max codebook Walsh-Hadamard transform SIMD <p>Every TurboQuant implementation we tested produces catastrophic output on GGUF quantized models. The reason is compound quantization error: weight quantization noise gets amplified through a second round of KV cache quantization, and softmax makes it worse. On Qwen 2.5 7B Q4_K_M, a representative Python implementation produces perplexity of 3,556 (baseline: 5.18). That is not a typo.</p> <p>This paper presents tq-kv, a Rust implementation of TurboQuant (ICLR 2026) that actually works on quantized models. We solve compound error through the 3-Fix Framework (sink token preservation, past-only quantizatioKV cache compressionn, cache state management) and introduce Pre-RoPE key quantization, which compresses keys before rotary position embedding to exploit position-independent channel statistics. Pre-RoPE reduces the perplexity gap by 34-59% at identical compression ratio. We also integrate KV Compaction, an attention-matching token reduction method orthogonal to bit compression, achieving combined compression up to 100-400x.</p> <p>On the systems side, we replace the dense QJL projection from the original paper with a structured Hadamard variant (SRHT-QJL) that is 115x faster and produces +4.5 dB better SNR. Fused attention scoring eliminates key decompression via AVX2+FMA centroid lookup, yielding 6-8.9x speedup. An O(1) incremental cache architecture cuts per-token overhead from 608 ms to 0.65 ms.</p> <p>We evaluate across five architectures (Llama-3, Qwen2.5, Gemma, Mistral, Phi-3) on both FP16 and GGUF Q4_K_M models. At 4-bit, tq-kv achieves +3.7% PPL delta with Pre-RoPE on GGUF (vs +17% for standard TQ and 3,556 for existing implementations), 9/9 needle-in-a-haystack pass rate, and 7.5-14.2x key compression. The system is 13.7K lines of Rust, ships as a crate on crates.io, and includes a complete inference engine with OpenAI-compatible API.</p> |
| title | TurboQuant on Quantized Models: Solving Compound Quantization Error with Pre-RoPE Compression and KV Compaction |
| topic | KV cache compression compound quantization error Pre-RoPE quantization KV compaction TurboQuant Rust GGUF Lloyd-Max codebook Walsh-Hadamard transform SIMD |
| url | https://doi.org/10.5281/zenodo.19375408 |