Cover Image

Finite-Time Analysis of Gradient Descent for Shallow Transformers

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Arda, Enes, Cayci, Semih, Eryilmaz, Atilla
Format:	Preprint
Published:	2026
Subjects:	Machine Learning Artificial Intelligence Optimization and Control
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Convergence of Gradient Descent for Recurrent Neural Networks: A Nonasymptotic Analysis
by: Cayci, Semih, et al.
Published: (2024)

Recurrent Natural Policy Gradient for POMDPs
by: Cayci, Semih, et al.
Published: (2024)

A Riemannian Optimization Perspective of the Gauss-Newton Method for Feedforward Neural Networks
by: Cayci, Semih
Published: (2024)

Convergence of Stochastic Gradient Langevin Dynamics in the Lazy Training Regime
by: Oberweis, Noah, et al.
Published: (2025)

Non-Asymptotic Optimization and Generalization Bounds for Stochastic Gauss-Newton in Overparameterized Models
by: Cayci, Semih
Published: (2025)

Linear Convergence of Entropy-Regularized Natural Policy Gradient with Linear Function Approximation
by: Cayci, Semih, et al.
Published: (2021)

Gradient Descent Efficiency Index
by: Dhingra, Aviral
Published: (2024)

Rod Flow: A Continuous-Time Model for Gradient Descent at the Edge of Stability
by: Regis, Eric, et al.
Published: (2026)

Almost Bayesian: The Fractal Dynamics of Stochastic Gradient Descent
by: Hennick, Max, et al.
Published: (2025)

On the Convergence of (Stochastic) Gradient Descent for Kolmogorov--Arnold Networks
by: Gao, Yihang, et al.
Published: (2024)

Weighted Low-rank Approximation via Stochastic Gradient Descent on Manifolds
by: Xu, Conglong, et al.
Published: (2025)

Enhancing Stochastic Gradient Descent: A Unified Framework and Novel Acceleration Methods for Faster Convergence
by: Deng, Yichuan, et al.
Published: (2024)

Multi-head Transformers Provably Learn Symbolic Multi-step Reasoning via Gradient Descent
by: Yang, Tong, et al.
Published: (2025)

Grams: Gradient Descent with Adaptive Momentum Scaling
by: Cao, Yang, et al.
Published: (2024)

Jacobian Descent for Multi-Objective Optimization
by: Quinton, Pierre, et al.
Published: (2024)

Gradient Descent Converges Linearly to Flatter Minima than Gradient Flow in Shallow Linear Networks
by: Beneventano, Pierfrancesco, et al.
Published: (2025)

Deterministic Policy Gradient for Reinforcement Learning with Continuous Time and State
by: Cheng, Ziheng, et al.
Published: (2025)

Asymptotic and Finite Sample Analysis of Nonexpansive Stochastic Approximations with Markovian Noise
by: Blaser, Ethan, et al.
Published: (2024)

A Single-Loop Gradient Descent and Perturbed Ascent Algorithm for Nonconvex Functional Constrained Optimization
by: Lu, Songtao
Published: (2022)

Towards Efficient Risk-Sensitive Policy Gradient: An Iteration Complexity Analysis
by: Liu, Rui, et al.
Published: (2024)

An Improved Finite-time Analysis of Temporal Difference Learning with Deep Neural Networks
by: Ke, Zhifa, et al.
Published: (2024)

On Finding Small Hyper-Gradients in Bilevel Optimization: Hardness Results and Improved Analysis
by: Chen, Lesi, et al.
Published: (2023)

Unraveling the Gradient Descent Dynamics of Transformers
by: Song, Bingqing, et al.
Published: (2024)

Achieving Tighter Finite-Time Rates for Heterogeneous Federated Stochastic Approximation under Markovian Sampling
by: Zhu, Feng, et al.
Published: (2025)

Delightful Policy Gradient
by: Osband, Ian
Published: (2026)

Fisher-Rao Gradient Flows of Linear Programs and State-Action Natural Policy Gradients
by: Müller, Johannes, et al.
Published: (2024)

Delightful Distributed Policy Gradient
by: Osband, Ian
Published: (2026)

Robust Control with Gradient Uncertainty
by: Qi, Qian
Published: (2025)

Asynchronous Distributed Reinforcement Learning for LQR Control via Zeroth-Order Block Coordinate Descent
by: Jing, Gangshan, et al.
Published: (2021)

Unbiased Gradient Low-Rank Projection
by: Pan, Rui, et al.
Published: (2025)

Decentralized Stochastic Gradient Descent Ascent for Finite-Sum Minimax Problems
by: Gao, Hongchang
Published: (2022)

Boosting Gradient Ascent for Continuous DR-submodular Maximization
by: Zhang, Qixin, et al.
Published: (2024)

Riemann Sum Optimization for Accurate Integrated Gradients Computation
by: Swain, Swadesh, et al.
Published: (2024)

Performative Policy Gradient: Optimality in Performative Reinforcement Learning
by: Basu, Debabrota, et al.
Published: (2025)

On the Convergence of Gradient Descent on Learning Transformers with Residual Connections
by: Qin, Zhen, et al.
Published: (2025)

Non-Smooth Weakly-Convex Finite-sum Coupled Compositional Optimization
by: Hu, Quanqi, et al.
Published: (2023)

Sven: Singular Value Descent as a Computationally Efficient Natural Gradient Method
by: Bright-Thonney, Samuel, et al.
Published: (2026)

A Unified Framework for Gradient Aggregation in Multi-Objective Optimization
by: Hu, Zeou, et al.
Published: (2026)

On the Optimal Construction of Unbiased Gradient Estimators for Zeroth-Order Optimization
by: Ma, Shaocong, et al.
Published: (2025)

Implicit Regularization of Gradient Flow on One-Layer Softmax Attention
by: Sheen, Heejune, et al.
Published: (2024)