Cover Image

Adaptive Sparse Softmax: An Effective and Efficient Softmax Variant

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Lv, Qi, Geng, Lei, Cao, Ziqiang, Cao, Min, Li, Sujian, Li, Wenjie, Fu, Guohong
Format:	Preprint
Published:	2025
Subjects:	Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

CGF-Softmax: A Cumulant-Based Softmax Reformulation for Efficient Inference under Homomorphic Encryption
by: Park, Hanjun, et al.
Published: (2026)

Softmax is not Enough (for Adaptive Conformal Classification)
by: Attar, Navid Akhavan, et al.
Published: (2026)

Vertex-Softmax: Tight Transformer Verification via Exact Softmax Optimization
by: Rezazadeh, Navid, et al.
Published: (2026)

Is Softmax Loss All You Need? A Principled Analysis of Softmax-family Loss
by: Pu, Yuanhao, et al.
Published: (2026)

Logit Dynamics in Softmax Policy Gradient Methods
by: Li, Yingru
Published: (2025)

Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts
by: Nguyen, Huy, et al.
Published: (2023)

Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?
by: Nguyen, Huy, et al.
Published: (2024)

Exploring the Frontiers of Softmax: Provable Optimization, Applications in Diffusion Model, and Beyond
by: Cao, Yang, et al.
Published: (2024)

To Softmax, or not to Softmax: that is the question when applying Active Learning for Transformer Models
by: Gonsior, Julius, et al.
Published: (2022)

Softmax-free Linear Transformers
by: Lu, Jiachen, et al.
Published: (2022)

Universal Approximation with Softmax Attention
by: Hu, Jerry Yao-Chieh, et al.
Published: (2025)

Softmax as a Lagrangian-Legendrian Seam
by: Lee-Jenkins, Christopher R.
Published: (2025)

Online Continual Learning via Logit Adjusted Softmax
by: Huang, Zhehao, et al.
Published: (2023)

Adaptive Sampled Softmax with Inverted Multi-Index: Methods, Theory and Applications
by: Chen, Jin, et al.
Published: (2025)

On the Invariants of Softmax Attention
by: Lee, Wonsuk
Published: (2026)

Scalable-Softmax Is Superior for Attention
by: Nakanishi, Ken M.
Published: (2025)

Gumbel-MPNN: Graph Rewiring with Gumbel-Softmax
by: Hoffmann, Marcel, et al.
Published: (2025)

Convergence Rates for Softmax Gating Mixture of Experts
by: Nguyen, Huy, et al.
Published: (2025)

NDCG-Consistent Softmax Approximation with Accelerated Convergence
by: Pu, Yuanhao, et al.
Published: (2025)

ITA: An Energy-Efficient Attention and Softmax Accelerator for Quantized Transformers
by: İslamoğlu, Gamze, et al.
Published: (2023)

Exploring the Impact of Temperature Scaling in Softmax for Classification and Adversarial Robustness
by: Xuan, Hao, et al.
Published: (2025)

Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts
by: Nguyen, Huy, et al.
Published: (2024)

Softmax pin to Cauchy-Poisson Study
by: Jones, Gareth
Published: (2026)

Softmax is not Enough (for Sharp Size Generalisation)
by: Veličković, Petar, et al.
Published: (2024)

The Information Geometry of Softmax: Probing and Steering
by: Park, Kiho, et al.
Published: (2026)

Softmax Transformers are Turing-Complete
by: Jiang, Hongjian, et al.
Published: (2025)

Hardware-Efficient Softmax and Layer Normalization with Guaranteed Normalization for Edge Devices
by: Choi, Dawon, et al.
Published: (2026)

On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Experts
by: Yan, Fanqi, et al.
Published: (2025)

Softpick: No Attention Sink, No Massive Activations with Rectified Softmax
by: Zuhri, Zayd M. K., et al.
Published: (2025)

On Least Square Estimation in Softmax Gating Mixture of Experts
by: Nguyen, Huy, et al.
Published: (2024)

Softmax Attention with Constant Cost per Token
by: Heinsen, Franz A.
Published: (2024)

Why Softmax Attention Outperforms Linear Attention
by: Deng, Yichuan, et al.
Published: (2023)

Fast Convergence of Softmax Policy Mirror Ascent
by: Asad, Reza, et al.
Published: (2024)

On the Expressiveness of Softmax Attention: A Recurrent Neural Network Perspective
by: Mongaras, Gabriel, et al.
Published: (2025)

Beyond Softmax: A Natural Parameterization for Categorical Random Variables
by: Manenti, Alessandro, et al.
Published: (2025)

Customizing the Inductive Biases of Softmax Attention using Structured Matrices
by: Kuang, Yilun, et al.
Published: (2025)

Binary Hypothesis Testing for Softmax Models and Leverage Score Models
by: Gu, Yuzhou, et al.
Published: (2024)

A Lyapunov Analysis of Softmax Policy Gradient for Stochastic Bandits
by: Lattimore, Tor
Published: (2026)

Rethinking Attention: Polynomial Alternatives to Softmax in Transformers
by: Saratchandran, Hemanth, et al.
Published: (2024)

Beyond Softmax: A New Perspective on Gradient Bandits
by: Melo, Emerson, et al.
Published: (2025)