Imagen de Portada

Routing Absorption in Sparse Attention: Why Random Gates Are Hard to Beat

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autor principal:	Aquino-Michaels, Keston
Formato:	Preprint
Publicado:	2026
Materias:	Machine Learning Computation and Language
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing
por: Piękos, Piotr, et al.
Publicado: (2025)

Random Initialization of Gated Sparse Adapters
por: Retault, Vi, et al.
Publicado: (2025)

Stochastic Attention: Connectome-Inspired Randomized Routing for Expressive Linear-Time Attention
por: Jin, Zehao, et al.
Publicado: (2026)

Why Softmax Attention Outperforms Linear Attention
por: Deng, Yichuan, et al.
Publicado: (2023)

Simulating Hard Attention Using Soft Attention
por: Yang, Andy, et al.
Publicado: (2024)

SEA: Sparse Linear Attention with Estimated Attention Mask
por: Lee, Heejun, et al.
Publicado: (2023)

The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs
por: Nawrot, Piotr, et al.
Publicado: (2025)

A Theory of Time-Sensitive Language Generation: Sparse Hallucination Beats Mode Collapse
por: Ganju, Atul, et al.
Publicado: (2026)

Gated Linear Attention Transformers with Hardware-Efficient Training
por: Yang, Songlin, et al.
Publicado: (2023)

PowerAttention: Exponentially Scaling of Receptive Fields for Effective Sparse Attention
por: Chen, Lida, et al.
Publicado: (2025)

In-context KV-Cache Eviction for LLMs via Attention-Gate
por: Zeng, Zihao, et al.
Publicado: (2024)

Block Sparse Flash Attention
por: Ohayon, Daniel, et al.
Publicado: (2025)

AdaSplash: Adaptive Sparse Flash Attention
por: Gonçalves, Nuno, et al.
Publicado: (2025)

Scaling Linear Attention with Sparse State Expansion
por: Pan, Yuqi, et al.
Publicado: (2025)

Hard-Attention Gates with Gradient Routing for Endoscopic Image Computing
por: Roffo, Giorgio, et al.
Publicado: (2024)

Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
por: Yuan, Jingyang, et al.
Publicado: (2025)

How Sparse Attention Approximates Exact Attention? Your Attention is Naturally $n^C$-Sparse
por: Deng, Yichuan, et al.
Publicado: (2024)

Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Eviction
por: He, Mutian, et al.
Publicado: (2025)

STS: Efficient Sparse Attention with Speculative Token Sparsity
por: Xu, Ceyu, et al.
Publicado: (2026)

AdaSplash-2: Faster Differentiable Sparse Attention
por: Gonçalves, Nuno, et al.
Publicado: (2026)

Sparse Attention across Multiple-context KV Cache
por: Cao, Ziyi, et al.
Publicado: (2025)

Unique Hard Attention: A Tale of Two Sides
por: Jerad, Selim, et al.
Publicado: (2025)

Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models
por: De, Soham, et al.
Publicado: (2024)

Sparse Query Attention (SQA): A Computationally Efficient Attention Mechanism with Query Heads Reduction
por: Filipek, Adam
Publicado: (2025)

From Sparse Dependence to Sparse Attention: Unveiling How Chain-of-Thought Enhances Transformer Sample Efficiency
por: Wen, Kaiyue, et al.
Publicado: (2024)

BiSparse-AAS: Bilinear Sparse Attention and Adaptive Spans Framework for Scalable and Efficient Text Summarization
por: Hagos, Desta Haileselassie, et al.
Publicado: (2025)

Energy-Gated Attention: Spectral Salience as an Inductive Bias for Transformer Attention
por: Zeris, Athanasios
Publicado: (2026)

ProxyAttn: Guided Sparse Attention via Representative Heads
por: Wang, Yixuan, et al.
Publicado: (2025)

LoLA: Low-Rank Linear Attention With Sparse Caching
por: McDermott, Luke, et al.
Publicado: (2025)

Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition
por: He, Zhengfu, et al.
Publicado: (2025)

Sparse Attention Remapping with Clustering for Efficient LLM Decoding on PIM
por: Fan, Zehao, et al.
Publicado: (2025)

DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention
por: Huang, Yuxiang, et al.
Publicado: (2026)

Universe Routing: Why Self-Evolving Agents Need Epistemic Control
por: Wang, Zhaohui Geoffrey
Publicado: (2026)

Forgetting Transformer: Softmax Attention with a Forget Gate
por: Lin, Zhixuan, et al.
Publicado: (2025)

NOSA: Native and Offloadable Sparse Attention
por: Huang, Yuxiang, et al.
Publicado: (2025)

HSR-Enhanced Sparse Attention Acceleration
por: Chen, Bo, et al.
Publicado: (2024)

SpecAttn: Speculating Sparse Attention
por: Shah, Harsh
Publicado: (2025)

Trainable Dynamic Mask Sparse Attention
por: Shi, Jingze, et al.
Publicado: (2025)

Energy-Gated Attention and Wavelet Positional Encoding: Complementary Inductive Biases for Transformer Attention
por: Zeris, Athanasios
Publicado: (2026)

IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse
por: Bai, Yushi, et al.
Publicado: (2026)