Cover Image

Universal Approximation with Softmax Attention

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Hu, Jerry Yao-Chieh, Liu, Hude, Chen, Hong-Yu, Wu, Weimin, Liu, Han
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Attention Mechanism, Max-Affine Partition, and Universal Approximation
by: Liu, Hude, et al.
Published: (2025)

Minimalist Softmax Attention Provably Learns Constrained Boolean Functions
by: Hu, Jerry Yao-Chieh, et al.
Published: (2025)

In-Context Algorithm Emulation in Fixed-Weight Transformers
by: Hu, Jerry Yao-Chieh, et al.
Published: (2025)

On Statistical Rates and Provably Efficient Criteria of Latent Diffusion Transformers (DiTs)
by: Hu, Jerry Yao-Chieh, et al.
Published: (2024)

Transformer Approximations from ReLUs
by: Hu, Jerry Yao-Chieh, et al.
Published: (2026)

On Statistical Rates of Conditional Diffusion Transformers: Approximation, Estimation and Minimax Optimality
by: Hu, Jerry Yao-Chieh, et al.
Published: (2024)

Are Hallucinations Bad Estimations?
by: Liu, Hude, et al.
Published: (2025)

MetaLA: Unified Optimal Linear Approximation to Softmax Attention Map
by: Chou, Yuhong, et al.
Published: (2024)

Provably Optimal Memory Capacity for Modern Hopfield Models: Transformer-Compatible Dense Associative Memories as Spherical Codes
by: Hu, Jerry Yao-Chieh, et al.
Published: (2024)

Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Models
by: Wu, Dennis, et al.
Published: (2024)

On Computational Limits of Modern Hopfield Models: A Fine-Grained Complexity Analysis
by: Hu, Jerry Yao-Chieh, et al.
Published: (2024)

POLO: Preference-Guided Multi-Turn Reinforcement Learning for Lead Optimization
by: Wang, Ziqing, et al.
Published: (2025)

Nonparametric Modern Hopfield Models
by: Hu, Jerry Yao-Chieh, et al.
Published: (2024)

Discrete Flow Matching Policy Optimization
by: Su, Maojiang, et al.
Published: (2026)

Fundamental Limits of Prompt Tuning Transformers: Universality, Capacity and Efficiency
by: Hu, Jerry Yao-Chieh, et al.
Published: (2024)

Outlier-Efficient Hopfield Layers for Large Transformer-Based Models
by: Hu, Jerry Yao-Chieh, et al.
Published: (2024)

A Theoretical Analysis of Discrete Flow Matching Generative Models
by: Su, Maojiang, et al.
Published: (2025)

Differentially Private Kernel Density Estimation
by: Liu, Erzhi, et al.
Published: (2024)

On Flow Matching KL Divergence
by: Su, Maojiang, et al.
Published: (2025)

Scalable-Softmax Is Superior for Attention
by: Nakanishi, Ken M.
Published: (2025)

BiSHop: Bi-Directional Cellular Learning for Tabular Data with Generalized Sparse Modern Hopfield Model
by: Xu, Chenwei, et al.
Published: (2024)

Computational Limits of Low-Rank Adaptation (LoRA) Fine-Tuning for Transformer Models
by: Hu, Jerry Yao-Chieh, et al.
Published: (2024)

STanHop: Sparse Tandem Hopfield Model for Memory-Enhanced Time Series Prediction
by: Wu, Dennis, et al.
Published: (2023)

Local Linear Attention: An Optimal Interpolation of Linear and Softmax Attention For Test-Time Regression
by: Zuo, Yifei, et al.
Published: (2025)

On Differentially Private String Distances
by: Hu, Jerry Yao-Chieh, et al.
Published: (2024)

On the Invariants of Softmax Attention
by: Lee, Wonsuk
Published: (2026)

Implicit Regularization of Gradient Flow on One-Layer Softmax Attention
by: Sheen, Heejune, et al.
Published: (2024)

Fast and Low-Cost Genomic Foundation Models via Outlier Removal
by: Luo, Haozheng, et al.
Published: (2025)

FLASH-D: FlashAttention with Hidden Softmax Division
by: Alexandridis, Kosmas, et al.
Published: (2025)

Forgetting Transformer: Softmax Attention with a Forget Gate
by: Lin, Zhixuan, et al.
Published: (2025)

Vertex-Softmax: Tight Transformer Verification via Exact Softmax Optimization
by: Rezazadeh, Navid, et al.
Published: (2026)

On the Mathematical Impossibility of Safe Universal Approximators
by: Yao, Jasper
Published: (2025)

Sigmoid Self-Attention has Lower Sample Complexity than Softmax Self-Attention: A Mixture-of-Experts Perspective
by: Yan, Fanqi, et al.
Published: (2025)

In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness
by: Collins, Liam, et al.
Published: (2024)

Softmax is not Enough (for Adaptive Conformal Classification)
by: Attar, Navid Akhavan, et al.
Published: (2026)

Softmax Linear Attention: Reclaiming Global Competition
by: Xu, Mingwei, et al.
Published: (2026)

Logit Dynamics in Softmax Policy Gradient Methods
by: Li, Yingru
Published: (2025)

DSL: Understanding and Improving Softmax Recommender Systems with Competition-Aware Scaling
by: Sahyouni, Bucher, et al.
Published: (2026)

In-Context Deep Learning via Transformer Models
by: Wu, Weimin, et al.
Published: (2024)

PSL: Rethinking and Improving Softmax Loss from Pairwise Perspective for Recommendation
by: Yang, Weiqin, et al.
Published: (2024)