Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing

Documenti analoghi

• SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention
  di: Csordás, Róbert, et al.
  Pubblicazione: (2023)
• The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs
  di: Nawrot, Piotr, et al.
  Pubblicazione: (2025)
• Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Eviction
  di: He, Mutian, et al.
  Pubblicazione: (2025)
• MoEUT: Mixture-of-Experts Universal Transformers
  di: Csordás, Róbert, et al.
  Pubblicazione: (2024)
• Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddings
  di: Gopalakrishnan, Anand, et al.
  Pubblicazione: (2025)