Image de couverture de livre

Scalable-Softmax Is Superior for Attention

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteur principal:	Nakanishi, Ken M.
Format:	Preprint
Publié:	2025
Sujets:	Computation and Language Artificial Intelligence Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Screening Is Enough
par: Nakanishi, Ken M.
Publié: (2026)

Forgetting Transformer: Softmax Attention with a Forget Gate
par: Lin, Zhixuan, et autres
Publié: (2025)

In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness
par: Collins, Liam, et autres
Publié: (2024)

To Softmax, or not to Softmax: that is the question when applying Active Learning for Transformer Models
par: Gonsior, Julius, et autres
Publié: (2022)

The Information Geometry of Softmax: Probing and Steering
par: Park, Kiho, et autres
Publié: (2026)

Scalable Efficient Training of Large Language Models with Low-dimensional Projected Attention
par: Lv, Xingtai, et autres
Publié: (2024)

DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention
par: Huang, Yuxiang, et autres
Publié: (2026)

Universal Approximation with Softmax Attention
par: Hu, Jerry Yao-Chieh, et autres
Publié: (2025)

Self-Correction Bench: Uncovering and Addressing the Self-Correction Blind Spot in Large Language Models
par: Tsui, Ken
Publié: (2025)

How Sparse Attention Approximates Exact Attention? Your Attention is Naturally $n^C$-Sparse
par: Deng, Yichuan, et autres
Publié: (2024)

Muon is Scalable for LLM Training
par: Liu, Jingyuan, et autres
Publié: (2025)

Attention Needs to Focus: A Unified Perspective on Attention Allocation
par: Fu, Zichuan, et autres
Publié: (2026)

Efficiently Dispatching Flash Attention For Partially Filled Attention Masks
par: Sharma, Agniv, et autres
Publié: (2024)

Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
par: Yuan, Jingyang, et autres
Publié: (2025)

Eigen Attention: Attention in Low-Rank Space for KV Cache Compression
par: Saxena, Utkarsh, et autres
Publié: (2024)

Depth-Recurrent Attention Mixtures: Giving Latent Reasoning the Attention it Deserves
par: Knupp, Jonas, et autres
Publié: (2026)

Why Softmax Attention Outperforms Linear Attention
par: Deng, Yichuan, et autres
Publié: (2023)

A Scalable Pipeline for Estimating Verb Frame Frequencies Using Large Language Models
par: Morgan, Adam M., et autres
Publié: (2025)

Attention Flows: Tracing LLM Conceptual Engagement via Story Summaries
par: Hicke, Rebecca M. M., et autres
Publié: (2026)

Scalable Chain of Thoughts via Elastic Reasoning
par: Xu, Yuhui, et autres
Publié: (2025)

SLR: Automated Synthesis for Scalable Logical Reasoning
par: Helff, Lukas, et autres
Publié: (2025)

Mode-Conditioning Unlocks Superior Test-Time Scaling
par: Wu, Chen Henry, et autres
Publié: (2025)

Scaling Reasoning without Attention
par: Zhao, Xueliang, et autres
Publié: (2025)

Higher-order Linear Attention
par: Zhang, Yifan, et autres
Publié: (2025)

Limitations of Normalization in Attention Mechanism
par: Mudarisov, Timur, et autres
Publié: (2025)

Block-Attention for Efficient Prefilling
par: Ma, Dongyang, et autres
Publié: (2024)

Selective Attention Improves Transformer
par: Leviathan, Yaniv, et autres
Publié: (2024)

Mixture of Attentions For Speculative Decoding
par: Zimmer, Matthieu, et autres
Publié: (2024)

NoMAD-Attention: Efficient LLM Inference on CPUs Through Multiply-add-free Attention
par: Zhang, Tianyi, et autres
Publié: (2024)

A Self-matching Training Method with Annotation Embedding Models for Ontology Subsumption Prediction
par: Shiraishi, Yukihiro, et autres
Publié: (2024)

Scaling Attention to Very Long Sequences in Linear Time with Wavelet-Enhanced Random Spectral Attention (WERSA)
par: Dentamaro, Vincenzo
Publié: (2025)

Scalable LLM Reasoning Acceleration with Low-rank Distillation
par: Dong, Harry, et autres
Publié: (2025)

Towards Scalable Oversight via Partitioned Human Supervision
par: Yin, Ren, et autres
Publié: (2025)

SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attention
par: Zhu, Qianchao, et autres
Publié: (2024)

Towards Scalable Automated Alignment of LLMs: A Survey
par: Cao, Boxi, et autres
Publié: (2024)

Dynamic Context Evolution for Scalable Synthetic Data Generation
par: Lingo, Ryan, et autres
Publié: (2026)

Imitating Language via Scalable Inverse Reinforcement Learning
par: Wulfmeier, Markus, et autres
Publié: (2024)

Efficient and Scalable Estimation of Tool Representations in Vector Space
par: Moon, Suhong, et autres
Publié: (2024)

From Self-Attention to Markov Models: Unveiling the Dynamics of Generative Transformers
par: Ildiz, M. Emrullah, et autres
Publié: (2024)

NOSA: Native and Offloadable Sparse Attention
par: Huang, Yuxiang, et autres
Publié: (2025)