:: Library Catalog

Cover Image

Saved in:

Bibliographic Details
Main Authors:	Zhang, Michael, Bhatia, Kush, Kumbong, Hermann, Ré, Christopher
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Computation and Language
Online Access:	https://arxiv.org/abs/2402.04347
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Cookbook: A framework for improving LLM generative abilities via programmatic data generating templates
by: Narayan, Avanika, et al.
Published: (2024)

LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
by: Zhou, Zikai, et al.
Published: (2025)

Why Softmax Attention Outperforms Linear Attention
by: Deng, Yichuan, et al.
Published: (2023)

Automated Rewards via LLM-Generated Progress Functions
by: Sarukkai, Vishnu, et al.
Published: (2024)

Kimi Linear: An Expressive, Efficient Attention Architecture
by: Kimi Team, et al.
Published: (2025)

The Expressive Power of Low Precision Softmax Transformers with (Summarized) Chain-of-Thought
by: Brösamle, Moritz, et al.
Published: (2026)

Stochastic Attention: Connectome-Inspired Randomized Routing for Expressive Linear-Time Attention
by: Jin, Zehao, et al.
Published: (2026)

RAM-Net: Expressive Linear Attention with Selectively Addressable Memory
by: Xiao, Kaicheng, et al.
Published: (2026)

Scalable-Softmax Is Superior for Attention
by: Nakanishi, Ken M.
Published: (2025)

Softmax Attention with Constant Cost per Token
by: Heinsen, Franz A.
Published: (2024)

Forgetting Transformer: Softmax Attention with a Forget Gate
by: Lin, Zhixuan, et al.
Published: (2025)

Evaluating the fairness of task-adaptive pretraining on unlabeled test data before few-shot text classification
by: Dubey, Kush
Published: (2024)

In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness
by: Collins, Liam, et al.
Published: (2024)

LoLCATs: On Low-Rank Linearizing of Large Language Models
by: Zhang, Michael, et al.
Published: (2024)

More Expressive Attention with Negative Weights
by: Lv, Ang, et al.
Published: (2024)

To Softmax, or not to Softmax: that is the question when applying Active Learning for Transformer Models
by: Gonsior, Julius, et al.
Published: (2022)

Softmax Transformers are Turing-Complete
by: Jiang, Hongjian, et al.
Published: (2025)

Taipan: Efficient and Expressive State Space Language Models with Selective Attention
by: Van Nguyen, Chien, et al.
Published: (2024)

The Information Geometry of Softmax: Probing and Steering
by: Park, Kiho, et al.
Published: (2026)

GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling
by: Dadgarnia, Alireza, et al.
Published: (2026)

Beyond Mimicry to Contextual Guidance: Knowledge Distillation for Interactive AI
by: Wang, Tong, et al.
Published: (2024)

SEA: Sparse Linear Attention with Estimated Attention Mask
by: Lee, Heejun, et al.
Published: (2023)

Linear Attention Sequence Parallelism
by: Sun, Weigao, et al.
Published: (2024)

Higher-order Linear Attention
by: Zhang, Yifan, et al.
Published: (2025)

On the Expressiveness of Softmax Attention: A Recurrent Neural Network Perspective
by: Mongaras, Gabriel, et al.
Published: (2025)

Trading Complexity for Expressivity Through Structured Generalized Linear Token Mixing
by: Fagnou, Erwan, et al.
Published: (2026)

Scaling Linear Attention with Sparse State Expansion
by: Pan, Yuqi, et al.
Published: (2025)

HMAR: Efficient Hierarchical Masked Auto-Regressive Image Generation
by: Kumbong, Hermann, et al.
Published: (2025)

Simple linear attention language models balance the recall-throughput tradeoff
by: Arora, Simran, et al.
Published: (2024)

Aioli: A Unified Optimization Framework for Language Model Data Mixing
by: Chen, Mayee F., et al.
Published: (2024)

Theoretical Constraints on the Expressive Power of $\mathsf{RoPE}$-based Tensor Attention Transformers
by: Li, Xiaoyu, et al.
Published: (2024)

Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Eviction
by: He, Mutian, et al.
Published: (2025)

Neural Attention Search Linear: Towards Adaptive Token-Level Hybrid Attention Models
by: Deng, Difan, et al.
Published: (2026)

Gated Linear Attention Transformers with Hardware-Efficient Training
by: Yang, Songlin, et al.
Published: (2023)

Counting Like Transformers: Compiling Temporal Counting Logic Into Softmax Transformers
by: Yang, Andy, et al.
Published: (2024)

The Expressive Capacity of State Space Models: A Formal Language Perspective
by: Sarrof, Yash, et al.
Published: (2024)

Unifying Linear-Time Attention via Latent Probabilistic Modelling
by: Dolga, Rares, et al.
Published: (2024)

Transformer Based Linear Attention with Optimized GPU Kernel Implementation
by: Gerami, Armin, et al.
Published: (2025)

LoLA: Low-Rank Linear Attention With Sparse Caching
by: McDermott, Luke, et al.
Published: (2025)

Probing Geometry of Next Token Prediction Using Cumulant Expansion of the Softmax Entropy
by: Viswanathan, Karthik, et al.
Published: (2025)