Image de couverture de livre

The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Zhang, Michael, Bhatia, Kush, Kumbong, Hermann, Ré, Christopher
Format:	Preprint
Publié:	2024
Sujets:	Machine Learning Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Cookbook: A framework for improving LLM generative abilities via programmatic data generating templates
par: Narayan, Avanika, et autres
Publié: (2024)

Why Softmax Attention Outperforms Linear Attention
par: Deng, Yichuan, et autres
Publié: (2023)

LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
par: Zhou, Zikai, et autres
Publié: (2025)

Automated Rewards via LLM-Generated Progress Functions
par: Sarukkai, Vishnu, et autres
Publié: (2024)

Kimi Linear: An Expressive, Efficient Attention Architecture
par: Kimi Team, et autres
Publié: (2025)

The Expressive Power of Low Precision Softmax Transformers with (Summarized) Chain-of-Thought
par: Brösamle, Moritz, et autres
Publié: (2026)

Stochastic Attention: Connectome-Inspired Randomized Routing for Expressive Linear-Time Attention
par: Jin, Zehao, et autres
Publié: (2026)

RAM-Net: Expressive Linear Attention with Selectively Addressable Memory
par: Xiao, Kaicheng, et autres
Publié: (2026)

Softmax Attention with Constant Cost per Token
par: Heinsen, Franz A.
Publié: (2024)

Scalable-Softmax Is Superior for Attention
par: Nakanishi, Ken M.
Publié: (2025)

Forgetting Transformer: Softmax Attention with a Forget Gate
par: Lin, Zhixuan, et autres
Publié: (2025)

Evaluating the fairness of task-adaptive pretraining on unlabeled test data before few-shot text classification
par: Dubey, Kush
Publié: (2024)

In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness
par: Collins, Liam, et autres
Publié: (2024)

More Expressive Attention with Negative Weights
par: Lv, Ang, et autres
Publié: (2024)

LoLCATs: On Low-Rank Linearizing of Large Language Models
par: Zhang, Michael, et autres
Publié: (2024)

To Softmax, or not to Softmax: that is the question when applying Active Learning for Transformer Models
par: Gonsior, Julius, et autres
Publié: (2022)

Taipan: Efficient and Expressive State Space Language Models with Selective Attention
par: Van Nguyen, Chien, et autres
Publié: (2024)

SEA: Sparse Linear Attention with Estimated Attention Mask
par: Lee, Heejun, et autres
Publié: (2023)

Linear Attention Sequence Parallelism
par: Sun, Weigao, et autres
Publié: (2024)

GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling
par: Dadgarnia, Alireza, et autres
Publié: (2026)

The Information Geometry of Softmax: Probing and Steering
par: Park, Kiho, et autres
Publié: (2026)

Softmax Transformers are Turing-Complete
par: Jiang, Hongjian, et autres
Publié: (2025)

Beyond Mimicry to Contextual Guidance: Knowledge Distillation for Interactive AI
par: Wang, Tong, et autres
Publié: (2024)

Higher-order Linear Attention
par: Zhang, Yifan, et autres
Publié: (2025)

On the Expressiveness of Softmax Attention: A Recurrent Neural Network Perspective
par: Mongaras, Gabriel, et autres
Publié: (2025)

Scaling Linear Attention with Sparse State Expansion
par: Pan, Yuqi, et autres
Publié: (2025)

Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Eviction
par: He, Mutian, et autres
Publié: (2025)

Neural Attention Search Linear: Towards Adaptive Token-Level Hybrid Attention Models
par: Deng, Difan, et autres
Publié: (2026)

Gated Linear Attention Transformers with Hardware-Efficient Training
par: Yang, Songlin, et autres
Publié: (2023)

Trading Complexity for Expressivity Through Structured Generalized Linear Token Mixing
par: Fagnou, Erwan, et autres
Publié: (2026)

Unifying Linear-Time Attention via Latent Probabilistic Modelling
par: Dolga, Rares, et autres
Publié: (2024)

Transformer Based Linear Attention with Optimized GPU Kernel Implementation
par: Gerami, Armin, et autres
Publié: (2025)

LoLA: Low-Rank Linear Attention With Sparse Caching
par: McDermott, Luke, et autres
Publié: (2025)

Theoretical Constraints on the Expressive Power of $\mathsf{RoPE}$-based Tensor Attention Transformers
par: Li, Xiaoyu, et autres
Publié: (2024)

Simple linear attention language models balance the recall-throughput tradeoff
par: Arora, Simran, et autres
Publié: (2024)

Don't Read Everything: A Curvature-Conditioned Query for Linear Attention
par: Le, Dong, et autres
Publié: (2026)

OSDN: Improving Delta Rule with Provable Online Preconditioning in Linear Attention
par: Zhou, Chenyu, et autres
Publié: (2026)

The Expressive Capacity of State Space Models: A Formal Language Perspective
par: Sarrof, Yash, et autres
Publié: (2024)

Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models
par: De, Soham, et autres
Publié: (2024)

Counting Like Transformers: Compiling Temporal Counting Logic Into Softmax Transformers
par: Yang, Andy, et autres
Publié: (2024)