Image de couverture de livre

Why Softmax Attention Outperforms Linear Attention

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Deng, Yichuan, Song, Zhao, Yuan, Kaijun, Zhou, Tianyi
Format:	Preprint
Publié:	2023
Sujets:	Computation and Language Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

How Sparse Attention Approximates Exact Attention? Your Attention is Naturally $n^C$-Sparse
par: Deng, Yichuan, et autres
Publié: (2024)

The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry
par: Zhang, Michael, et autres
Publié: (2024)

Scalable-Softmax Is Superior for Attention
par: Nakanishi, Ken M.
Publié: (2025)

Softmax Attention with Constant Cost per Token
par: Heinsen, Franz A.
Publié: (2024)

Forgetting Transformer: Softmax Attention with a Forget Gate
par: Lin, Zhixuan, et autres
Publié: (2025)

Neural Attention Search Linear: Towards Adaptive Token-Level Hybrid Attention Models
par: Deng, Difan, et autres
Publié: (2026)

In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness
par: Collins, Liam, et autres
Publié: (2024)

SEA: Sparse Linear Attention with Estimated Attention Mask
par: Lee, Heejun, et autres
Publié: (2023)

Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix
par: Liang, Yingyu, et autres
Publié: (2024)

Linear Attention Sequence Parallelism
par: Sun, Weigao, et autres
Publié: (2024)

Causal Attention with Lookahead Keys
par: Song, Zhuoqing, et autres
Publié: (2025)

Training Tensor Attention Efficiently: From Cubic to Almost Linear Time
par: Cao, Yang, et autres
Publié: (2024)

Stochastic Attention: Connectome-Inspired Randomized Routing for Expressive Linear-Time Attention
par: Jin, Zehao, et autres
Publié: (2026)

Kimi Linear: An Expressive, Efficient Attention Architecture
par: Kimi Team, et autres
Publié: (2025)

RoPE Attention Can Be Trained in Almost Linear Time
par: Cao, Yang, et autres
Publié: (2024)

Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Eviction
par: He, Mutian, et autres
Publié: (2025)

Attention Needs to Focus: A Unified Perspective on Attention Allocation
par: Fu, Zichuan, et autres
Publié: (2026)

Scaling Linear Attention with Sparse State Expansion
par: Pan, Yuqi, et autres
Publié: (2025)

OSDN: Improving Delta Rule with Provable Online Preconditioning in Linear Attention
par: Zhou, Chenyu, et autres
Publié: (2026)

AttnCache: Accelerating Self-Attention Inference for LLM Prefill via Attention Cache
par: Song, Dinghong, et autres
Publié: (2025)

Routing Absorption in Sparse Attention: Why Random Gates Are Hard to Beat
par: Aquino-Michaels, Keston
Publié: (2026)

Higher-order Linear Attention
par: Zhang, Yifan, et autres
Publié: (2025)

Gated Linear Attention Transformers with Hardware-Efficient Training
par: Yang, Songlin, et autres
Publié: (2023)

RAM-Net: Expressive Linear Attention with Selectively Addressable Memory
par: Xiao, Kaicheng, et autres
Publié: (2026)

Unifying Linear-Time Attention via Latent Probabilistic Modelling
par: Dolga, Rares, et autres
Publié: (2024)

Transformer Based Linear Attention with Optimized GPU Kernel Implementation
par: Gerami, Armin, et autres
Publié: (2025)

LoLA: Low-Rank Linear Attention With Sparse Caching
par: McDermott, Luke, et autres
Publié: (2025)

NoMAD-Attention: Efficient LLM Inference on CPUs Through Multiply-add-free Attention
par: Zhang, Tianyi, et autres
Publié: (2024)

Optimizing Mixture of Block Attention
par: Xiao, Guangxuan, et autres
Publié: (2025)

Immunogenicity Prediction with Dual Attention Enables Vaccine Target Selection
par: Li, Song, et autres
Publié: (2024)

Don't Read Everything: A Curvature-Conditioned Query for Linear Attention
par: Le, Dong, et autres
Publié: (2026)

One Pass Streaming Algorithm for Super Long Token Attention Approximation in Sublinear Space
par: Addanki, Raghav, et autres
Publié: (2023)

Multi-Layer Attention is the Amplifier of Demonstration Effectiveness
par: Wang, Dingzirui, et autres
Publié: (2025)

Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models
par: De, Soham, et autres
Publié: (2024)

QiMeng-Attention: SOTA Attention Operator is generated by SOTA Attention Algorithm
par: Zhou, Qirui, et autres
Publié: (2025)

Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models
par: Kunstner, Frederik, et autres
Publié: (2024)

HSR-Enhanced Sparse Attention Acceleration
par: Chen, Bo, et autres
Publié: (2024)

Why Any-Order Autoregressive Models Need Two-Stream Attention: A Structural-Semantic Tradeoff
par: Pynadath, Patrick, et autres
Publié: (2026)

Scaling Attention to Very Long Sequences in Linear Time with Wavelet-Enhanced Random Spectral Attention (WERSA)
par: Dentamaro, Vincenzo
Publié: (2025)

Exploring Public Attention in the Circular Economy through Topic Modelling with Twin Hyperparameter Optimisation
par: Song, Junhao, et autres
Publié: (2024)