Exploiting Sparsity for Long Context Inference: Million Token Contexts on Commodity GPUs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916609750204416 |
|---|---|
| author | Synk, Ryan Hoover, Monte Kirchenbauer, John Jain, Neel Stein, Alex Shu, Manli Sanchez, Josue Melendez Duraiswami, Ramani Goldstein, Tom |
| author_facet | Synk, Ryan Hoover, Monte Kirchenbauer, John Jain, Neel Stein, Alex Shu, Manli Sanchez, Josue Melendez Duraiswami, Ramani Goldstein, Tom |
| contents | There is growing demand for performing inference with hundreds of thousands of input tokens on trained transformer models. Inference at this extreme scale demands significant computational resources, hindering the application of transformers at long contexts on commodity (i.e not data center scale) hardware. To address the inference time costs associated with running self-attention based transformer language models on long contexts and enable their adoption on widely available hardware, we propose a tunable mechanism that reduces the cost of the forward pass by attending to only the most relevant tokens at every generation step using a top-k selection mechanism. We showcase the efficiency gains afforded by our method by performing inference on context windows up to 1M tokens using approximately 16GB of GPU RAM. Our experiments reveal that models are capable of handling the sparsity induced by the reduced number of keys and values. By attending to less than 2% of input tokens, we achieve over 95% of model performance on common benchmarks (RULER, AlpacaEval, and Open LLM Leaderboard). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_06766 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Exploiting Sparsity for Long Context Inference: Million Token Contexts on Commodity GPUs Synk, Ryan Hoover, Monte Kirchenbauer, John Jain, Neel Stein, Alex Shu, Manli Sanchez, Josue Melendez Duraiswami, Ramani Goldstein, Tom Computation and Language There is growing demand for performing inference with hundreds of thousands of input tokens on trained transformer models. Inference at this extreme scale demands significant computational resources, hindering the application of transformers at long contexts on commodity (i.e not data center scale) hardware. To address the inference time costs associated with running self-attention based transformer language models on long contexts and enable their adoption on widely available hardware, we propose a tunable mechanism that reduces the cost of the forward pass by attending to only the most relevant tokens at every generation step using a top-k selection mechanism. We showcase the efficiency gains afforded by our method by performing inference on context windows up to 1M tokens using approximately 16GB of GPU RAM. Our experiments reveal that models are capable of handling the sparsity induced by the reduced number of keys and values. By attending to less than 2% of input tokens, we achieve over 95% of model performance on common benchmarks (RULER, AlpacaEval, and Open LLM Leaderboard). |
| title | Exploiting Sparsity for Long Context Inference: Million Token Contexts on Commodity GPUs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.06766 |