Image de couverture de livre

Implicit Bias and Fast Convergence Rates for Self-attention

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Vasudeva, Bhavya, Deora, Puneesh, Thrampoulidis, Christos
Format:	Preprint
Publié:	2024
Sujets:	Machine Learning Optimization and Control
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

On the Optimization and Generalization of Multi-head Attention
par: Deora, Puneesh, et autres
Publié: (2023)

In-Context Occam's Razor: How Transformers Prefer Simpler Hypotheses on the Fly
par: Deora, Puneesh, et autres
Publié: (2025)

How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Data
par: Vasudeva, Bhavya, et autres
Publié: (2025)

Understanding Contextual Recall in Transformers: How Finetuning Enables In-Context Reasoning over Pretraining Knowledge
par: Vasudeva, Bhavya, et autres
Publié: (2026)

The Rich and the Simple: On the Implicit Bias of Adam and SGD
par: Vasudeva, Bhavya, et autres
Publié: (2025)

Implicit Bias of Spectral Descent and Muon on Multiclass Separable Data
par: Fan, Chen, et autres
Publié: (2025)

Facts in Stats: Impacts of Pretraining Diversity on Language Model Generalization
par: Behnia, Tina, et autres
Publié: (2025)

Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamics
par: Garrod, Connall, et autres
Publié: (2025)

Implicit Bias and Convergence of Matrix Stochastic Mirror Descent
par: Akhtiamov, Danil, et autres
Publié: (2026)

Convergence and Implicit Bias of Gradient Descent on Continual Linear Classification
par: Jung, Hyunji, et autres
Publié: (2025)

Nonsmooth Implicit Differentiation: Deterministic and Stochastic Convergence Rates
par: Grazzi, Riccardo, et autres
Publié: (2024)

Memory capacity of two layer neural networks with smooth activations
par: Madden, Liam, et autres
Publié: (2023)

FedCluster: Boosting the Convergence of Federated Learning via Cluster-Cycling
par: Chen, Cheng, et autres
Publié: (2020)

Non-Parametric Learning of Stochastic Differential Equations with Non-asymptotic Fast Rates of Convergence
par: Bonalli, Riccardo, et autres
Publié: (2023)

Improving Generalization and Convergence by Enhancing Implicit Regularization
par: Wang, Mingze, et autres
Publié: (2024)

Implicit Bias of Mirror Flow on Separable Data
par: Pesme, Scott, et autres
Publié: (2024)

Convergence Rate Analysis of LION
par: Dong, Yiming, et autres
Publié: (2024)

Next-token prediction capacity: general upper bounds and a lower bound for transformers
par: Madden, Liam, et autres
Publié: (2024)

Implicit Bias of Gradient Descent for Non-Homogeneous Deep Networks
par: Cai, Yuhang, et autres
Publié: (2025)

Transformers as Support Vector Machines
par: Tarzanagh, Davoud Ataee, et autres
Publié: (2023)

Implicit Bias of AdamW: $\ell_\infty$ Norm Constrained Optimization
par: Xie, Shuo, et autres
Publié: (2024)

Towards The Implicit Bias on Multiclass Separable Data Under Norm Constraints
par: Xie, Shengping, et autres
Publié: (2026)

MAP Estimation with Denoisers: Convergence Rates and Guarantees
par: Pesme, Scott, et autres
Publié: (2025)

Constant Stepsize Q-learning: Distributional Convergence, Bias and Extrapolation
par: Zhang, Yixuan, et autres
Publié: (2024)

Implicit Bias in Matrix Factorization and its Explicit Realization in a New Architecture
par: Hou, Yikun, et autres
Publié: (2025)

Open Problem: Anytime Convergence Rate of Gradient Descent
par: Kornowski, Guy, et autres
Publié: (2024)

Incremental Gauss--Newton Methods with Superlinear Convergence Rates
par: Zhou, Zhiling, et autres
Publié: (2024)

Robust Sublinear Convergence Rates for Iterative Bregman Projections
par: Peyré, Gabriel
Publié: (2026)

Limits of Convergence-Rate Control for Open-Weight Safety
par: Rosati, Domenic, et autres
Publié: (2026)

Improved Convergence Rates of Muon Optimizer for Nonconvex Optimization
par: Nagashima, Shuntaro, et autres
Publié: (2026)

Convergence Rate of the Last Iterate of Stochastic Proximal Algorithms
par: Vaidyan, Kevin Kurian Thomas, et autres
Publié: (2026)

On the Implicit Bias of Adam
par: Cattaneo, Matias D., et autres
Publié: (2023)

Reusing Historical Trajectories in Natural Policy Gradient via Importance Sampling: Convergence and Convergence Rate
par: Lin, Yifan, et autres
Publié: (2024)

Convergence of Implicit Gradient Descent for Training Two-Layer Physics-Informed Neural Networks
par: Xu, Xianliang, et autres
Publié: (2024)

The Implicit Bias of Heterogeneity towards Invariance: A Study of Multi-Environment Matrix Sensing
par: Xu, Yang, et autres
Publié: (2024)

Incremental Quasi-Newton Methods with Faster Superlinear Convergence Rates
par: Liu, Zhuanghua, et autres
Publié: (2024)

Adaptive Algorithms with Sharp Convergence Rates for Stochastic Hierarchical Optimization
par: Gong, Xiaochuan, et autres
Publié: (2025)

Adam-HNAG: A Convergent Reformulation of Adam with Accelerated Rate
par: Yu, Yaxin, et autres
Publié: (2026)

Sharper Convergence Rates for Nonconvex Optimisation via Reduction Mappings
par: Markou, Evan, et autres
Publié: (2025)

Theoretical Analysis on how Learning Rate Warmup Accelerates Convergence
par: Liu, Yuxing, et autres
Publié: (2025)