Image de couverture de livre

Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Piękos, Piotr, Csordás, Róbert, Schmidhuber, Jürgen
Format:	Preprint
Publié:	2025
Sujets:	Machine Learning Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention
par: Csordás, Róbert, et autres
Publié: (2023)

The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs
par: Nawrot, Piotr, et autres
Publié: (2025)

Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Eviction
par: He, Mutian, et autres
Publié: (2025)

MoEUT: Mixture-of-Experts Universal Transformers
par: Csordás, Róbert, et autres
Publié: (2024)

Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddings
par: Gopalakrishnan, Anand, et autres
Publié: (2025)

Routing Absorption in Sparse Attention: Why Random Gates Are Hard to Beat
par: Aquino-Michaels, Keston
Publié: (2026)

SEA: Sparse Linear Attention with Estimated Attention Mask
par: Lee, Heejun, et autres
Publié: (2023)

Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
par: Yuan, Jingyang, et autres
Publié: (2025)

How Sparse Attention Approximates Exact Attention? Your Attention is Naturally $n^C$-Sparse
par: Deng, Yichuan, et autres
Publié: (2024)

Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts
par: Zhang, Zeliang, et autres
Publié: (2024)

Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts
par: Ahrac, Sagi, et autres
Publié: (2026)

CoSMoEs: Compact Sparse Mixture of Experts
par: Huber, Patrick, et autres
Publié: (2025)

Block Sparse Flash Attention
par: Ohayon, Daniel, et autres
Publié: (2025)

Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models
par: Wang, Zihan, et autres
Publié: (2025)

SLA2: Sparse-Linear Attention with Learnable Routing and QAT
par: Zhang, Jintao, et autres
Publié: (2026)

SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts
par: Muzio, Alexandre, et autres
Publié: (2024)

LoLA: Low-Rank Linear Attention With Sparse Caching
par: McDermott, Luke, et autres
Publié: (2025)

ProxyAttn: Guided Sparse Attention via Representative Heads
par: Wang, Yixuan, et autres
Publié: (2025)

PowerAttention: Exponentially Scaling of Receptive Fields for Effective Sparse Attention
par: Chen, Lida, et autres
Publié: (2025)

AdaSplash: Adaptive Sparse Flash Attention
par: Gonçalves, Nuno, et autres
Publié: (2025)

Scaling Linear Attention with Sparse State Expansion
par: Pan, Yuqi, et autres
Publié: (2025)

DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention
par: Huang, Yuxiang, et autres
Publié: (2026)

Self-Organising Neural Discrete Representation Learning à la Kohonen
par: Irie, Kazuki, et autres
Publié: (2023)

Omni-Router: Sharing Routing Decisions in Sparse Mixture-of-Experts for Speech Recognition
par: Gu, Zijin, et autres
Publié: (2025)

Sparse Attention across Multiple-context KV Cache
par: Cao, Ziyi, et autres
Publié: (2025)

STS: Efficient Sparse Attention with Speculative Token Sparsity
par: Xu, Ceyu, et autres
Publié: (2026)

AdaSplash-2: Faster Differentiable Sparse Attention
par: Gonçalves, Nuno, et autres
Publié: (2026)

NOSA: Native and Offloadable Sparse Attention
par: Huang, Yuxiang, et autres
Publié: (2025)

SpecAttn: Speculating Sparse Attention
par: Shah, Harsh
Publié: (2025)

Trainable Dynamic Mask Sparse Attention
par: Shi, Jingze, et autres
Publié: (2025)

HSR-Enhanced Sparse Attention Acceleration
par: Chen, Bo, et autres
Publié: (2024)

DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
par: Song, Chenyang, et autres
Publié: (2026)

IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse
par: Bai, Yushi, et autres
Publié: (2026)

Sparse Query Attention (SQA): A Computationally Efficient Attention Mechanism with Query Heads Reduction
par: Filipek, Adam
Publié: (2025)

Maximum Score Routing For Mixture-of-Experts
par: Dong, Bowen, et autres
Publié: (2025)

BiSparse-AAS: Bilinear Sparse Attention and Adaptive Spans Framework for Scalable and Efficient Text Summarization
par: Hagos, Desta Haileselassie, et autres
Publié: (2025)

From Sparse Dependence to Sparse Attention: Unveiling How Chain-of-Thought Enhances Transformer Sample Efficiency
par: Wen, Kaiyue, et autres
Publié: (2024)

Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition
par: He, Zhengfu, et autres
Publié: (2025)

Sparse Attention Remapping with Clustering for Efficient LLM Decoding on PIM
par: Fan, Zehao, et autres
Publié: (2025)

Sparse Attention Decomposition Applied to Circuit Tracing
par: Franco, Gabriel, et autres
Publié: (2024)