Why Attend to Everything? Focus is the Key
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910175497027584 |
|---|---|
| author | Yao, Hengshuai Chen, Xing Murtadha, Ahmed Li, Jin Yadkori, Yasin Abbasi Shao, Shuai Liu, Changling Wang, Guan Yuan, Mingli Chen, William Song, Sen |
| author_facet | Yao, Hengshuai Chen, Xing Murtadha, Ahmed Li, Jin Yadkori, Yasin Abbasi Shao, Shuai Liu, Changling Wang, Guan Yuan, Mingli Chen, William Song, Sen |
| contents | Standard attention scales quadratically with sequence length. Efficient attention methods reduce this O(n^2) cost, but when retrofitted into pretrained models, they often degrade perplexity, downstream accuracy, or both. We introduce Focus, a method that learns which token pairs matter. Focus adds a small set of learnable centroids--as few as 148K parameters per layer--that act as gates: only token pairs belonging to the same centroid group attend to each other over long ranges. Focus is composable: it can be added to any pretrained model by training only the centroids while keeping all original weights frozen.
Experiments show that composing Focus onto pretrained models yields zero degradation on downstream benchmarks across model sizes from 124M to 70B parameters and five attention architectures. Surprisingly, sparse Focus attention outperforms full attention at 124M scale (30.3 vs. 31.4 perplexity) and matches full attention when trained from scratch at 7B scale (13.82 vs. 13.89). Focus is also fast: top-k group membership gives a 2x speedup with better quality than the original pretrained model. Using our FlashAttention decomposition, Focus achieves an 8.6x speedup at 1M tokens without custom kernels. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_03260 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Why Attend to Everything? Focus is the Key Yao, Hengshuai Chen, Xing Murtadha, Ahmed Li, Jin Yadkori, Yasin Abbasi Shao, Shuai Liu, Changling Wang, Guan Yuan, Mingli Chen, William Song, Sen Computation and Language Artificial Intelligence Standard attention scales quadratically with sequence length. Efficient attention methods reduce this O(n^2) cost, but when retrofitted into pretrained models, they often degrade perplexity, downstream accuracy, or both. We introduce Focus, a method that learns which token pairs matter. Focus adds a small set of learnable centroids--as few as 148K parameters per layer--that act as gates: only token pairs belonging to the same centroid group attend to each other over long ranges. Focus is composable: it can be added to any pretrained model by training only the centroids while keeping all original weights frozen. Experiments show that composing Focus onto pretrained models yields zero degradation on downstream benchmarks across model sizes from 124M to 70B parameters and five attention architectures. Surprisingly, sparse Focus attention outperforms full attention at 124M scale (30.3 vs. 31.4 perplexity) and matches full attention when trained from scratch at 7B scale (13.82 vs. 13.89). Focus is also fast: top-k group membership gives a 2x speedup with better quality than the original pretrained model. Using our FlashAttention decomposition, Focus achieves an 8.6x speedup at 1M tokens without custom kernels. |
| title | Why Attend to Everything? Focus is the Key |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2604.03260 |